התמלול המלא של Build More Reliable Distributed Systems By Breaking Them With Jepsen

falling over anytime you tried to list keys in the database. So there was there's a lot of rough and tumble action in those days. 36 00:04:38.254 --> 00:04:45.794 Nowadays, I think people take partitions, crashes, that sort of thing more seriously. I see a lot of folks at least trying to do some degree of testing for those. 37 00:04:46.500 --> 00:04:52.120 And yet, when I do consulting work, often I'll find databases where it looks like nobody 38 00:04:52.980 --> 00:04:54.840 tried to verify behavior 39 00:04:55.335 --> 00:04:56.155 during vaults. 40 00:04:56.694 --> 00:05:04.550 And aside from Jepsen, what are some of the other methods that people typically use for stress testing testing or evaluating 41 00:05:04.930 --> 00:05:06.710 the guarantees of these systems? 42 00:05:07.010 --> 00:05:13.134 I look at Jepson, because it's an experimental approach as being 1 part of a complete breakfast, a continuum of quality 43 00:05:13.514 --> 00:05:14.014 methods. 44 00:05:15.354 --> 00:05:18.094 1 of those is to start with a a proven algorithm, some sort of abstract model of your system, and 45 00:05:21.540 --> 00:05:30.600 either to have a hand or a machine check proof of it or at least a model checker, maybe with TLA. And if you are lucky, you can do some sort of code extraction with Coq or 46 00:05:31.505 --> 00:05:34.725 high level or sorry, for f h o l with Isabelle. 47 00:05:35.105 --> 00:05:42.860 But that's that's not, I think, how most production systems get built. Most systems get built by pointing at a whiteboard and writing a bunch of code. 48 00:05:43.240 --> 00:05:50.300 And then your tools for safety analysis tend to be more along the lines of unit tests and integration tests. 49 00:05:51.145 --> 00:05:53.325 Jepsen takes a black box approach, which 50 00:05:53.705 --> 00:05:55.885 puts it sort of at the far end of 51 00:05:56.425 --> 00:06:00.125 integration testing. Other systems like Cassandra's d test suite, 52 00:06:00.550 --> 00:06:01.770 take a similar approach. 53 00:06:02.070 --> 00:06:06.169 You'll see people who do sort of manual testing where they do fault injection via 54 00:06:06.710 --> 00:06:10.795 somebody just running commands on the cluster or toggling some sort of hardware 55 00:06:11.175 --> 00:06:15.515 power switch or network switch. That's an approach that's taken by FoundationDB. 56 00:06:15.895 --> 00:06:19.675 FoundationDB also does a lot of automated simulation testing where they will 57 00:06:20.020 --> 00:06:24.440 run a virtualized network with many simulated nodes in the same process, and then they can 58 00:06:24.820 --> 00:06:47.035 intercept messages and reorder them to get interesting results. So I think that there's there's a lot of approaches people take to testing systems. Jepsen tries to general and to to work across many types of systems, which often means that it's not as deep or as capable as exploring the state space as, say, a simulation test that's written specifically for the database that you're working on might be. And then 59 00:06:47.415 --> 00:06:50.715 as far as some of the real world usage 60 00:06:51.200 --> 00:06:55.620 of things like databases and other mission critical distributed systems, 61 00:06:55.920 --> 00:07:05.305 what are some of the real world impacts that can occur as a result of these failures that aren't properly vetted or properly guarded against? 62 00:07:05.685 --> 00:07:07.160 Oh, you can get all kinds of faults. 63 00:07:07.720 --> 00:07:13.260 Obviously, data could be lost. So you you write something, and then it just disappears later. You can have 64 00:07:13.640 --> 00:07:15.180 different nodes that compete 65 00:07:15.715 --> 00:07:19.735 on some version of the history. So maybe 1 person thinks that the 66 00:07:20.035 --> 00:07:23.815 the list you're adding things to is 1, 2, 3. The other thing's it's 4, 5, 6. 67 00:07:24.350 --> 00:07:29.890 You can have temporal anomalies where you read some information from the past instead of the present. 68 00:07:30.510 --> 00:07:37.604 You could have transactional violations where 2 transactions kind of land on top of each other or interleave in some weird way. And all these systems, 69 00:07:38.065 --> 00:07:39.365 that I've tested exhibit, 70 00:07:40.384 --> 00:07:42.485 typically 1 or more of these behaviors. 71 00:07:42.900 --> 00:07:46.120 The question is sort of how hard do you have to push them to get there. 72 00:07:46.500 --> 00:08:04.099 And then in terms of the actual testing approach of using Jepsen for being able to find some of these error modes that exist in a particular system, what is your overall approach for designing the test suite and customizing it for a given system? Because 73 00:08:04.560 --> 00:08:11.125 all of these different platforms will have different guarantees that they're trying to provide or different approaches to linearizability 74 00:08:11.585 --> 00:08:12.325 or serializability 75 00:08:12.785 --> 00:08:23.250 or the strictness of the isolation that they provide. And I know that that provides a fairly extensive surface area to try and cover in any test suite. And I'm wondering just what your 76 00:08:23.615 --> 00:08:24.355 approach is 77 00:08:25.215 --> 00:08:30.540 for determining what are the key points to interact with, and then also any variability in the 78 00:08:36.940 --> 00:08:38.320 operations that are being performed. 79 00:08:38.855 --> 00:08:41.435 Yeah. You're you're quite right. There's there's sort of 2 80 00:08:41.735 --> 00:08:47.195 huge surface areas to cover. 1 of them is the breadth of the API, all the different things databases do. 81 00:08:47.920 --> 00:08:51.300 Modern SQL databases have so many functions and so many 82 00:08:51.680 --> 00:08:56.180 interacting components that it's really difficult to predict, you know, how they're going to interact. 83 00:08:56.915 --> 00:08:57.415 And 84 00:08:58.195 --> 00:09:01.574 it also is tricky because when you get optimizers into the mix, 85

00:09:01.954 --> 00:09:05.014 you know, depending on something like the size of your data, 86 00:09:05.329 --> 00:09:09.910 you might have 2 very different code paths that get involved with different safety properties. 87 00:09:11.490 --> 00:09:21.634 So I don't have a good answer for that. You know, I I write as many tests as I can in the time that I have, but that's usually just a tiny fraction of the overall API. My guiding principle is to look for 88 00:09:22.080 --> 00:09:30.420 operations which are general and which will be sort of the most difficult for the system to do safely. As an example, there is a property called the consensus number, 89 00:09:30.925 --> 00:09:34.065 which sort of tells you how difficult it is to achieve 90 00:09:35.005 --> 00:09:37.025 a sort of operation in an atomic, 91 00:09:37.645 --> 00:09:38.785 way using consensus. 92 00:09:39.620 --> 00:09:41.720 So when you have a database which 93 00:09:42.660 --> 00:09:45.960 offers, say, reads and writes over a register 94 00:09:46.715 --> 00:09:53.775 or offers compare and set operations, it turns out that the compare and set operation is actually more difficult to provide safely 95 00:09:54.630 --> 00:09:56.570 than just reads and writes over registers. 96 00:09:56.950 --> 00:10:01.370 And so I'll test compare and set to try and stress the system more. If I have 97 00:10:02.625 --> 00:10:13.709 an operation that works like appending something to a set or to a list, well, the set is order independent, whereas the list retains order. So I'll try to do something on a list because it will expose more possible, 98 00:10:14.410 --> 00:10:18.990 histories in which the outcome can discriminate between correct and incorrect executions, 99 00:10:19.395 --> 00:10:36.579 if that makes any sense. Yeah. That definitely makes good sense. Because as you said, the ordering is explicit in those types of data structures. And when you have these multiple concurrent clients that are interacting and you're dealing with a system that is trying to provide maybe snapshot isolation, 100 00:10:36.959 --> 00:10:38.145 you ensure that 101 00:10:39.425 --> 00:10:50.029 the right and then the subsequent read produces the expected result given the point of time and the other clients that you're connecting with. And that also leads me to wonder how you 102 00:10:50.570 --> 00:10:51.470 sort of synchronize 103 00:10:51.930 --> 00:11:11.220 the the state of those multiple clients to be able to determine what the correct output should be at the end of a test suite to be able to say, I'm writing all of these values. It's being run-in this multi threaded fashion. And when everything is said and done, this is what the actual output should be, assuming that everything is running properly. And then also 104 00:11:11.644 --> 00:11:14.065 particularly for eventually consistent systems 105 00:11:14.524 --> 00:11:15.024 identifying 106 00:11:15.485 --> 00:11:16.144 how they're 107 00:11:16.524 --> 00:11:19.425 resolving those potential conflicts that are being 108 00:11:19.970 --> 00:11:21.830 created by those multiple clients? 109 00:11:22.209 --> 00:11:23.670 Yes. Yes. Obviously, 110 00:11:24.290 --> 00:11:33.975 real world problems with databases are typically concurrent problems, and Jepsen tests concurrently. So we have to be able to understand exactly what the concurrency structure was 111 00:11:34.355 --> 00:11:37.560 of all the clients that are interacting with the system. To do that, Jepsen 112 00:11:37.860 --> 00:11:39.320 puts them all in the same, 113 00:11:40.100 --> 00:11:41.960 VM, the same JVM process, 114 00:11:42.820 --> 00:11:44.120 which means that we have 115 00:11:44.605 --> 00:11:58.890 a local sequentially consistent store that we can use to record the actions of each client, and that tells us what began first. But, of course, there's still concurrency there. Even though we have tight bounds on it, 2 transactions are likely running at the same time. And that means that when you're trying to 116 00:11:59.270 --> 00:12:08.975 verify the system is correct or not, you have to consider both possible orders. You know, 1 transaction executing first or the other transaction executing first. Writing efficient checkers for those 117 00:12:09.275 --> 00:12:24.755 permutations of operations turns out to be really tricky. Eventually, consistent systems, like you mentioned, actually have this nice property, typically, that they should commute, that lots of different orders are legal. Any any possible ordering should be okay. And so when you test eventually consistent systems, it's often 118 00:12:25.375 --> 00:12:36.840 a little bit easier to design the test for because all you need to do is apply the operations in 1 order, and they should be equivalent to any other order, at least for something like a CRDT. If it's not, well, then you have a combinatorial explosion again. 119 00:12:37.265 --> 00:12:37.765 And 120 00:12:38.225 --> 00:12:44.245 given the massive surface area of the problem demand that you're trying to validate, what is your 121 00:12:44.545 --> 00:12:47.660 heuristic for determining the overall completeness 122 00:12:47.960 --> 00:12:54.620 of your test suite or determining a useful definition of done for the purposes of a particular engagement? 123 00:12:55.235 --> 00:13:04.695 Right. Right. It's it's such a huge problem. And the fact that any of this works at all is sort of astonishing. 1 of the things that I use to guide my work is the idea of 124 00:13:05.380 --> 00:13:08.760 a falsifiable result. I want to design a test which 125 00:13:09.140 --> 00:13:09.640 yields 126 00:13:10.420 --> 00:13:11.560 a failing result 127 00:13:11.940 --> 00:13:12.920 before it passes. 128 00:13:13.605 --> 00:13:19.464 And I should be able to turn up and down a concurrency knob or turn on and off some feature in the database. 129 00:13:19.765 --> 00:13:31.655 Say, I I enable the linearizable mode and it should pass linearizability tests.

And if I disable it, it should fail those tests. If it still passes with linearizability disabled, I know the test wasn't powerful enough to find some fault. So that's that's 1 notion of completeness. 130 00:13:32.195 --> 00:13:39.320 Another is just looking at the API and kind of staring at the algorithm and trying to guess all the ways in which it could go wrong and then designing 131 00:13:39.779 --> 00:13:49.384 faults for those. But, ultimately, the space of faults is very large, and we're never gonna be able to cover all of it meaningfully. The only reason this works is because the state space is often degenerate 132 00:13:49.685 --> 00:13:52.425 and that many different permutations of failures 133 00:13:53.080 --> 00:13:57.500 yield similar outcomes. Another metric I use to guide my work is, 134 00:13:58.520 --> 00:13:59.020 I 135 00:13:59.480 --> 00:14:01.825 watch the logs and I look at visualizations 136 00:14:02.285 --> 00:14:03.105 of the history, 137 00:14:03.645 --> 00:14:06.065 very carefully. And I try to intuit 138 00:14:06.605 --> 00:14:12.740 have the behaviors that I have introduced with the system, the network partitions, the process crashes, 139 00:14:13.120 --> 00:14:14.100 have those faults 140 00:14:14.560 --> 00:14:19.675 created some sort of phase change in the system? Do I see a characteristic shift in latency, or, 141 00:14:20.135 --> 00:14:29.440 do I see a transition between returning linearizable and nonlinearizable results even if both are legal? I wanna make sure that those transitions occur because that tells me my test is actually 142 00:14:29.980 --> 00:14:30.480 measuring 143 00:14:31.420 --> 00:14:36.545 some sort of event in the system, like a like a failover or promotion of a new node or some sort of reconciliation 144 00:14:36.925 --> 00:14:40.225 process. If I don't see that happen, I have a hint that my system isn't 145 00:14:40.605 --> 00:14:43.505 being tested rigorously enough. And then another element 146 00:14:43.910 --> 00:14:44.649 of complexity 147 00:14:45.110 --> 00:14:50.970 in dealing with testing these types of systems is being able to actually get them deployed in some representative 148 00:14:51.785 --> 00:14:52.285 distribution 149 00:14:52.665 --> 00:14:59.725 of making sure that there are multiple instances running, that they're all communicating properly and clustered together. And then for 150 00:15:07.019 --> 00:15:18.610 the curious what your experience has been as far as being able to do this in a reasonable fashion without having to do everything manually, particularly given the different ways of 151 00:15:18.990 --> 00:15:24.824 deployment and sort of different levels of maturity in terms of the operability of these systems and the configuration 152 00:15:25.285 --> 00:15:27.545 availability and things like that? Yeah. 153 00:15:28.165 --> 00:15:33.200 Luckily, most computer systems are meant to be deployed on mixed computers over IP, 154 00:15:34.080 --> 00:15:49.535 which is

good because that's how Jepsen works. But, you know, as as modern deployments show up, like like serverless things or hosted services, those are much harder for me to test because I don't have control over their infrastructure. The good news is that when something is meant to be installed, I can typically 155 00:15:50.080 --> 00:15:54.980 write some sort of automation for it, like downloading a package and installing it and, 156 00:15:55.360 --> 00:16:17.975 running some shell commands to configure it. All of that is automated by Jepsen. So Jepsen has a a whole deploy system built into it, which pushes out different binaries and run shell commands and uploads files and gets the cluster running. Running that automation is the first part of any test process. And because of the fact that the primary focus of Jepson is on the distributed systems guarantees, 157 00:16:18.915 --> 00:16:24.290 another level of complexity in getting these system set up properly is dealing with things like encryption 158 00:16:24.750 --> 00:16:26.449 and authentication and authorization. 159 00:16:26.829 --> 00:16:32.785 I'm assuming that you just completely leave that to the side so that you can focus on the durability and serializability 160 00:16:33.085 --> 00:16:37.585 guarantees of the system. Yeah. I typically ignore those, because they're often orthogonal. 161 00:16:38.045 --> 00:16:50.295 It's typically the case There's, like, an authentication layer at the top end of the database when you first connect. And once you're past that internally, it's just the same function calls you would get either way. That's not always true. Some systems, like in cryptocurrencies 162 00:16:51.315 --> 00:17:01.500 will have a sort of signature verification, like, baked into the very core of the consensus protocol. And there, it makes sense to test sort of adversarial methods. But generally with Jepsen, 163 00:17:01.959 --> 00:17:02.940 you look for 164 00:17:03.399 --> 00:17:05.980 getting the most bang for your buck. You look for the key 165 00:17:06.684 --> 00:17:10.625 couple of functions of the the basic data model, which will give you 166 00:17:11.485 --> 00:17:16.860 the best insight into how the concurrency control mechanism works as a whole, And then hopefully that generalizes. 167 00:17:17.320 --> 00:17:21.259 So you might miss data type specific or authentication specific problems, 168 00:17:21.639 --> 00:17:28.285 but those things could be incrementally added later if you have time to add those tests. And then for the actual process of 169 00:17:28.665 --> 00:17:30.125 building the deployment 170 00:17:30.425 --> 00:17:30.925 and 171 00:17:31.225 --> 00:17:58.590 running the overall workflow of setting up the tests, suite, executing it, and then evaluating the output. Can you just talk through that whole process and some of the challenges that you face, particularly at the end in terms of interpreting the results and being able to provide useful feedback to the people

who are building these systems and engaging with you to find errors that they need to correct? Yeah. My overall process, when I quote clients, I typically say there's a there's a 4 week minimum, 172 00:17:59.065 --> 00:18:13.150 to get a report out the door. And that gives me enough time to get the system installed, which usually takes the first couple days to a week to get it installed reliably. Turns out a lot of systems, when you install them, don't necessarily set up reliably the first time. 173 00:18:13.770 --> 00:18:25.325 Then there's a lot of documentation reading that usually takes a couple days. And I try to understand the model of the system, what kind of API calls would be involved, what kind of invariance to be measured. Then I will design the 174 00:18:25.809 --> 00:18:35.865 the workload and the client that that interprets that workload and actually goes and makes network calls and gets results and understands them. And then once that's done, I'll start introducing faults. And that usually is 175 00:18:36.404 --> 00:18:38.745 the the kind of process of weeks 176 00:18:39.205 --> 00:18:42.504 23 is adding different faults and running the tests. 177 00:18:42.870 --> 00:18:44.650 The actual test workflow is 178 00:18:45.190 --> 00:18:47.850 you run a shell command, jumps and starts. It 179 00:18:48.150 --> 00:18:49.930 installs all the stuff on the nodes, 180 00:18:50.365 --> 00:18:52.625 starts making requests, introduces faults. 181 00:18:52.925 --> 00:18:58.625 Once it's recorded a history for a few minutes or a few hours, it will analyze that history and spit out some, 182 00:18:59.200 --> 00:19:02.240 files and some textual output. So I'll read those those, 183 00:19:02.640 --> 00:19:14.740 explanations of what happened. I will look at the graphs it produces. I'll look at the charts and try to interpret those and and, tell the people who built the database what I found. And in terms of Jepson itself, how is it implemented, 184 00:19:22.684 --> 00:19:39.820 originally, Jepsen was built quite quickly, just to demonstrate a couple of bugs in in 1 or 2 particular systems. There's an early prototype in Erlang, which was actually intended to be a proxy that would sit between all of your nodes and, disassemble their network flows and and, cut them together in new orders. That was a lovely idea, but it turns out that writing, 185 00:19:40.715 --> 00:19:44.335 protocol level decoders for every database is a really tricky idea. 186 00:19:44.715 --> 00:19:48.254 And, it was much more interesting and effective to just focus on running systems, 187 00:19:48.634 --> 00:19:53.980 sort of in their native environment. So round 2 of Jepsen is a a JVM program, which 188 00:19:54.440 --> 00:20:04.515 would run some basic workloads and requests against the server concurrently. And then I did automation, all the setup, introducing faults, separately using a deploy tool I wrote called Soltasid. 189 00:20:04.870 --> 00:20:08.650 So I would be manually

sitting at the keyboard triggering a fault and watching what happens. 190 00:20:09.110 --> 00:20:12.570 Round 3 of Jepsen, which I think started around 2014, 2015, 191 00:20:12.894 --> 00:20:15.075 brought those things together and said 192 00:20:15.455 --> 00:20:23.850 the operating system and database deployment tools should be an integral part of the test. So when you're writing a Jepsen test now, you define 193 00:20:24.230 --> 00:20:27.530 what kind of operating system do you wanna use, how do you set up your database, 194 00:20:27.965 --> 00:20:38.919 what kind of schedule do you want to perform, how should that schedule be applied to the database and the results recorded, And then how do you wanna analyze that abstract history? And that decoupling between 195 00:20:39.299 --> 00:20:50.485 this abstract model of the world, like, write 3 to the register named x and read the current value of register y and observe 2. That abstract world is what the 196 00:20:50.785 --> 00:20:56.190 analysis tools in Jepsen are written to understand, which means that you can reuse the same analyzers from many different databases. 197 00:20:57.130 --> 00:21:08.910 A lot of the interpretive work in Jepsen is now saying, well, what does it mean to write a register in FooDB? How how are how am I gonna encode this particular abstract operation? And often there are many different ways to do it. 198 00:21:11.550 --> 00:21:15.170 So I'll write lots of different families of tests, which encode that sort of abstract operation differently in the database itself. 199 00:21:15.470 --> 00:21:36.920 It sounds like from your description of actually running through the workflow that Jepsen actually has components that sit on the database nodes themselves in order to be able to communicate back and forth and execute these operations and record different data points. And so given that it is, at least on some level, a distributed system in itself, and I'm curious if you have created a Jepsen test for Jepsen. 200 00:21:37.885 --> 00:21:49.300 It's a it's a good idea. But, there's no agent. There's no local processes involved in Jepsen, to try to keep things as simple as possible. Everything runs in 1 bog standard JVM program on your local node. 201 00:21:49.920 --> 00:21:52.665 All interaction remotely is done for SSH or, 202 00:21:53.065 --> 00:21:54.765 Kubernetes or Docker interfaces. 203 00:21:55.305 --> 00:21:57.245 Another level of complexity 204 00:21:57.785 --> 00:22:04.000 in some of these systems is that some of the failure modes may only trigger when there are certain volumes of data 205 00:22:04.300 --> 00:22:10.170 in the system or if you have a large volume of data in a cluster and 1 of the nodes goes down and then it creates a cascading failure of trying to rebalance information and then 206 00:22:11.965 --> 00:22:15.745 failure of trying to rebalance information and then introducing 207 00:22:16.045 --> 00:22:26.030 new data into the cluster. And I'm curious if

you explore those types of problem spaces as well. Yeah. This is something that Jepson is really bad at. Jepson excels at this kind of sweet spot of 208 00:22:26.330 --> 00:22:27.310 moderate concurrency, 209 00:22:28.264 --> 00:22:29.565 short temporal duration, 210 00:22:30.024 --> 00:22:31.325 medium sized datasets. 211 00:22:32.024 --> 00:22:35.800 We're talking about histories of maybe a 100000 to a 1000000 operations tops. 212 00:22:36.360 --> 00:22:39.740 But it's very true that a lot of times, systems only fall over 213 00:22:40.360 --> 00:22:41.500 for large datasets 214 00:22:42.040 --> 00:22:45.180 or for super high concurrencies or super high node counts. 215 00:22:45.555 --> 00:22:47.015 Those are important to test, 216 00:22:47.315 --> 00:22:56.750 but they're the difficulty in testing those systems with something like Jepson is that trying to analyze the history for correctness might be computationally intractable. 217 00:22:57.130 --> 00:23:05.755 It could take you millennia maybe to analyze it depending on the property you're trying to look at. So I think what often makes sense is to run Jepsen as a part of a suite of tests 218 00:23:06.135 --> 00:23:13.690 or or Jepsen like system. You know, if you if you have, distributed safety testing Cassandra, for example, that exists in conjunction with other tests, which do 219 00:23:14.150 --> 00:23:15.210 long run times, 220 00:23:15.590 --> 00:23:16.650 high data volumes, 221 00:23:17.030 --> 00:23:19.450 but don't try to do rigorous safety analysis. 222 00:23:19.865 --> 00:23:31.669 They're just asking, you know, does the system continue to respond to requests and is latency okay? They're not trying to understand was it linearizable because that problem is NP hard. And then as far as the overall 223 00:23:32.049 --> 00:23:49.730 pros and cons that you've seen of building Jepsen and Closure, I'm curious what you have found in that regard as far as the accessibility of the system and making it easy for other people to be able to use Jepsen for their own purposes or contribute new tests for new distributed systems? Yeah. 224 00:23:50.370 --> 00:23:51.730 You know, choosing a list, 225 00:23:52.050 --> 00:23:52.950 is not necessarily 226 00:23:53.250 --> 00:23:56.950 the idea you wanna adopt if you're trying to get a big community. 227 00:23:57.315 --> 00:24:01.495 Although, I've done it twice now. Riemann and Jepsen converted systems administrators 228 00:24:01.875 --> 00:24:03.895 and database engineers to Closurists, 229 00:24:04.915 --> 00:24:07.890 strangely enough. And when I teach classes on Jepsen, 230 00:24:08.270 --> 00:24:22.995 I find that people are usually able to get closure within a few hours and at least write some sort of basic test following along. You know, obviously, it takes longer to become a fluent engineer, but the language isn't necessarily the barrier that I initially thought it was. In terms of suitability 231 00:24:23.295 --> 00:24:23.795 for, 232 00:24:24.660 --> 00:24:28.679 somebody who's proficient with a language, is Closure the right fit for this problem? 233 00:24:29.059 --> 00:24:30.440 I think that the JVM 234 00:24:30.740 --> 00:24:36.365 is a good candidate for testing databases in general. So doing something in Java, Scala, Groovy, Kotlin, 235 00:24:36.825 --> 00:24:37.325 Closure, 236 00:24:37.705 --> 00:24:39.805 it's nice because you have access to JDBC. 237 00:24:40.345 --> 00:25:00.605 Almost every database publishes some sort of Java client of some type. And as long as you have interop between those languages and the JVM, you can call those clients. So from Closure, I can call anybody's Java client, and that generally works pretty well. You also want to have reasonable performance. It doesn't have to be super fast. You wanna be able to go somewhat quick, and you want real threads. So that rules out, 238 00:25:01.010 --> 00:25:03.910 a single thread at runtime. Like JavaScript, it would rule out, 239 00:25:04.370 --> 00:25:16.705 you know, a slower language like Ruby or Python. You wanna have something more on the side of Haskell or c or Rust or the JVM, something with a a reasonable profiler and some attention paid to VM performance. 240 00:25:17.140 --> 00:25:24.360 And then because this work is experimental, I think you want a language which is concise and expressive. 1 of the pieces of feedback I get from people 241 00:25:24.735 --> 00:25:25.554 when I teach, 242 00:25:26.255 --> 00:25:27.315 Jepsen in classes 243 00:25:27.855 --> 00:25:41.100 is I'm really confused by all the parentheses, which is, of course, the classic Lisp critique. It's it's absolutely valid. It's a new way of doing syntax, and it's confusing. But people really like that Jepsen's API is so compact and that it is easy to 244 00:25:41.505 --> 00:25:43.845 read, understand, and explore new schedules. 245 00:25:44.225 --> 00:25:45.685 So I feel like the API design 246 00:25:46.225 --> 00:25:56.025 has worked out well there. And if I were to design a new version of Jepsen of the language, I would look for something which makes it easy to write compact code. And you mentioned that there are some considerations 247 00:26:02.505 --> 00:26:10.240 any other elements that you would either redesign or re architect if you were to start over today and build a similar set of tooling. 248 00:26:10.595 --> 00:26:15.095 Yeah. At the risk of grabbing the traditional third rail of of language arguments, 249 00:26:15.475 --> 00:26:16.934 a type system might be nice. 250 00:26:18.040 --> 00:26:20.060 Obviously, closure is a dynamic 251 00:26:20.760 --> 00:26:23.740 semi unit type language with protocols and 252 00:26:24.280 --> 00:26:29.655 basically JVM's dynamic type checks underneath the hood. But there are some instances in Jepsen where 253 00:26:30.275 --> 00:26:40.820 you're gonna manipulate a whole bunch of data which has really similar representations for something with the same name. Like, a node might be a

string, an internal identifier, 254 00:26:41.120 --> 00:26:42.179 a node number, 255 00:26:42.480 --> 00:26:43.299 a handle, 256 00:26:43.760 --> 00:26:44.419 a client, 257 00:26:44.720 --> 00:26:48.475 and you wanna discriminate between those things reliably. Having a type system 258 00:26:49.015 --> 00:26:54.640 helps there. So I've been using core dot typed in some parts of Jepsen for places where it's hard to keep those things straight. 259 00:26:55.360 --> 00:26:58.260 But it might be nice to write it in something like, 260 00:26:58.720 --> 00:27:03.395 Kotlin or Haskell, where I could have some sort of nominal typing instead of, 261 00:27:03.875 --> 00:27:07.335 just structural. I think in terms of architecture changes that I would make to Jepson, 262 00:27:07.715 --> 00:27:08.535 1 of the 263 00:27:09.235 --> 00:27:13.320 problems that I had for a long time with Jepson was the generator system, 264 00:27:14.279 --> 00:27:17.179 the part that schedules operations and emits concurrent 265 00:27:17.639 --> 00:27:31.680 things to do for the database. That system was initially built as a mutable object that every thread, every worker in the system would ask for an operation. So each thread says, hey, generator. Give me something to do. And the generator would, you know, mutate some state and spit out our op. 266 00:27:33.020 --> 00:27:46.035 Internally, if you wanted to do something like wait for 2 minutes, you would just call thread sleep, and then the thread scheduler would take care of making sure that the thread went to sleep and woke up at the right time. This is intuitive and it's easy to write, but 267 00:27:46.340 --> 00:27:56.165 it can lead to really tricky cases of deadlock. Because if you want to interrupt a thread that's sleeping or a thread that's engaged in some sort of barrier where it's blocking awaiting other threads, 268 00:27:56.725 --> 00:28:04.345 you run the risk of maybe double interrupting and knocking a thread out of the interrupt recovery code itself. It becomes devilishly hard to write correctly. 269 00:28:05.580 --> 00:28:08.559 The other big problem I had with the generators was that 270 00:28:08.940 --> 00:28:10.000 they weren't responsive 271 00:28:10.380 --> 00:28:14.080 to things that happened in the world. So the generator was kind of this 272 00:28:14.615 --> 00:28:24.110 pure source of information. It told you what to do, but it didn't take any feedback. And that meant that you couldn't write something like, try writing values until you see at least 1 success. 273 00:28:24.670 --> 00:28:36.765 And that's an important thing to do if you're gonna be testing database. You might wanna say, like, keep reading until you actually get something so that your test means something at the end. The way that you would work around that limitation would be to have some mutable state. So you would have the client 274 00:28:37.225 --> 00:28:39.405 and the generator share a piece of mutable 275 00:28:39.920 --> 00:28:53.735 information, some variable that they would both update and read in order to change the generator behavior. But that introduces this kind of ugly coupling, and it was confusing for users. So I've redesigned the generator system as a purely functional effects system with an interpreter. 276 00:28:54.035 --> 00:29:05.309 I thought that this was gonna be a terrible idea, and it may yet prove to be. But so far, it seems to be better. It means that we can do all the mutation in a single thread, which eliminates all of the thread scheduling ugliness 277 00:29:05.615 --> 00:29:10.195 and deadlocks that you have with the original generator system. And it also means that you can take a feedback. 278 00:29:10.575 --> 00:29:21.040 The generator can be updated with each operation's completion as well indication. This might be way more into the weeds than you ought to talk about, but it's my my big architecture change in Jepsen for the time being. Yeah. It's always interesting 279 00:29:21.554 --> 00:29:22.054 getting 280 00:29:22.355 --> 00:29:28.695 behind the scenes of the problems that exist in a particular tool. Because as a user, it's easy to 281 00:29:28.995 --> 00:29:31.320 gloss over those things or not understand 282 00:29:31.620 --> 00:29:32.840 the problem space, 283 00:29:33.539 --> 00:29:52.920 effectively enough to be able to know what is wrong with a given implementation. But as somebody who spent so much time in the code, there are always going to be warts or things that you don't like or things that you would do differently. And it's interesting to get some of the perspective on that. You know, it is it's funny. Jepsen's architecture has been basically unchanged for about 5 years. I take backwards compatibility really seriously. 284 00:29:53.300 --> 00:30:22.010 And, this generator change just went live about a month or 2 ago, and it was the biggest kind of break I've made to Jepsen's API. But even then, it doesn't change the structure of tests. It's still a generator. It still has the same decomposition. And I feel like this is the 3rd iteration, and it has been reasonably stable. It it does the right it's the right way to break up the problem. But there's a lot of experimentation and bad design choices that I'm still making in things like the the checkers, the bits that verify histories and tell you if there are property problems. 285 00:30:22.310 --> 00:30:24.090 So I am currently 286 00:30:25.510 --> 00:30:27.770 knee deep in the weeds on a 287 00:30:28.075 --> 00:30:31.135 a new checker based on l for doing set 288 00:30:31.515 --> 00:30:39.300 verification and transactional systems. And that puppy, I have no idea how to write it correctly. This is my 3rd attempt. Maybe this time it will work. 289 00:30:40.720 --> 00:30:42.740 That still tends to be a never ending process. 290 00:30:43.205 --> 00:30:44.745 As somebody who has been 291 00:30:45.125 --> 00:30:46.745 using and working 292 00:30:47.365 --> 00:31:00.670 on Jepsen for quite

a while now, and you have worked with a number of different companies and open source projects on various types of distributed systems. So you have possibly the best perspective in terms of the 293 00:31:01.115 --> 00:31:09.419 overall problem space and the ways that it's being tackled. But I'm wondering what you see as some of the most common failure modes 294 00:31:09.799 --> 00:31:20.035 that exist in platforms and some of the ones that continue to crop up and that continue to be overlooked maybe in initial attempts at building some of these types of platforms? 295 00:31:20.655 --> 00:31:25.555 Yeah. And this has shifted over time. The industry is getting a lot better at building distributed systems. 296 00:31:26.300 --> 00:31:28.960 This is a little tricky. There is a class of 297 00:31:29.340 --> 00:31:33.760 what you might call an error where people rely on the well ordering 298 00:31:34.115 --> 00:31:43.940 or the k ordering of clocks. They assume the clocks are roughly on time. They track time at the same rate, and then they use that to build, say, leases for leaders. 299 00:31:45.139 --> 00:32:01.495 This is safe and correct so long as the clocks are correct. But we don't have a lot of empirical data on how often clocks are correct and how badly they go wrong. I would like to see more of that. In the meantime, it seems valid that you would write a database based on these assumptions and make the clock skew thresholds tunable. 300 00:32:02.679 --> 00:32:11.260 But there's this sort of uncomfortable feeling. They're like, well, eventually, the clocks go wrong, and then what happens to my data? Am I okay with that? So this is 301 00:32:12.184 --> 00:32:18.684 sort of an active field development. And I think all people who are choosing to do clock stuff at this point databases like, 302 00:32:19.710 --> 00:32:20.610 TIdb, GigabyteDB, 303 00:32:21.149 --> 00:32:21.649 CockroachDB, 304 00:32:21.950 --> 00:32:24.510 I think. It's all blurring together at this point. 305 00:32:26.110 --> 00:32:29.635 They they all rely on some clocks, I believe, for leader leases in different ways, 306 00:32:30.115 --> 00:32:38.295 And they're making that choice in an informed way. They know that the clocks could be bad. So I don't think that's necessarily an error. It's it's maybe more, 307 00:32:38.755 --> 00:32:39.415 a choice. 308 00:32:40.100 --> 00:32:44.520 1 of the more serious errors that I see nowadays tends to be around understanding indeterminate 309 00:32:44.900 --> 00:32:45.400 results. 310 00:32:45.860 --> 00:32:48.520 So oftentimes, there'd be some sort of RPC mechanism 311 00:32:49.005 --> 00:32:53.025 inside of a database where 1 component is supposed to send a message to another and get a response. 312 00:32:53.485 --> 00:33:04.715 And if it doesn't get a response it expects, that's an error. But a lot of times, people will interpret that error as meaning that the operation did not take place, and it is therefore safe to retry it. 313 00:33:05.115 --> 00:33:13.054 Sometimes it's safe to retry operations. But sometimes if you retry them, you wind up introducing some sort of anomaly, like a double execution or violating 314 00:33:14.050 --> 00:33:15.190 transactional isolation. 315 00:33:15.970 --> 00:33:20.390 So several of the bugs that I encountered recently had to do with internal retrying mechanisms 316 00:33:20.934 --> 00:33:26.075 or client level retrying mechanisms, which didn't understand that they weren't allowed to retry in that context. 317 00:33:26.774 --> 00:33:28.154 Tracking that stuff safely 318 00:33:28.615 --> 00:33:45.125 is something that I talk about with most of my clients. And most of them understand it, and they're trying to do it. But it's hard to keep track of all the different ways in which things can fail because there's just so many different, you know, terrible things that computers can do to you. Yeah. Computers are wonderful and awful all at the same time, 319 00:33:45.585 --> 00:33:56.675 and generally more awful than not for people who are trying to actually build them and work on them. Oh, for sure. I have so much respect for for distributed systems engineers, people who are trying to build databases because it is this 320 00:33:57.075 --> 00:34:00.535 terribly complicated problem where you have to think of everything, and then 321 00:34:00.915 --> 00:34:04.615 here I come like some sort of rampant toddler knocking over all the blocks. 322 00:34:08.060 --> 00:34:17.174 Yeah. And yet they pay you to go and break things for them and tell them what they did wrong. I am weirdly effective at it. I don't know. The computers around me seem to break more than for other people. 323 00:34:17.635 --> 00:34:19.174 I'd in fact, I'm currently 324 00:34:19.714 --> 00:34:24.440 in some sort of bureaucratic hellscape because of some sort of bank computer prop for the company. 325 00:34:26.820 --> 00:34:27.220 And, 326 00:34:28.180 --> 00:34:30.840 it seems like every week, I discover some sort of new, 327 00:34:31.194 --> 00:34:40.414 you know, fresh way that computers have have gone terribly wrong. I wasn't able to use my browser for, like, 2 weeks last week. Well, perhaps when you're done with distributed systems research, you can move on to security. 328 00:34:41.130 --> 00:34:43.630 Yeah, please. Don't don't give me ideas. 329 00:34:44.170 --> 00:34:45.710 And then in terms of 330 00:34:46.730 --> 00:34:48.030 the problems that you identify, 331 00:34:48.330 --> 00:34:55.265 what are some of the classes of issues that you have found to be the most difficult to resolve in some of these distributed systems? 332 00:34:55.645 --> 00:35:04.390 The ones that are difficult to fix once you understand them are generally architectural problems where it's like, oh, we have to rip apart the whole thing and replace it with a new algorithm. 333 00:35:05.170 --> 00:35:06.355 But I do think that 334 00:35:06.915 --> 00:35:12.615 folks are using more and more proven

algorithms nowadays. I see a lot of use of Raft. I see more deployment of CRDTs. 335 00:35:12.915 --> 00:35:14.215 I see people using Paxos, 336 00:35:15.155 --> 00:35:15.895 and that's, 337 00:35:16.970 --> 00:35:29.585 that eliminates a whole class of errors where people just made up some sort of consensus system and then hope that it worked. The thing that that we don't know how to do yet is to build transactional protocols on top of those individual 338 00:35:30.045 --> 00:35:31.025 consensus systems. 339 00:35:31.565 --> 00:35:35.185 And there's a lot of active research and a lot of industrial effort 340 00:35:35.660 --> 00:35:38.320 going into that. And I think that's a place where people will make 341 00:35:38.940 --> 00:35:39.920 sort of tricky 342 00:35:40.540 --> 00:35:41.040 mistakes, 343 00:35:43.025 --> 00:35:48.484 which is, again, not to say that the engineers doing this are bad. It's that they're tackling an almost impossible problem with a huge, 344 00:35:48.944 --> 00:35:50.964 you know, failure space and are 345 00:35:51.290 --> 00:35:52.830 encountering the kind of obvious 346 00:35:53.210 --> 00:35:57.070 difficulties in solving that impossible problem. And what are some of the 347 00:35:57.530 --> 00:35:59.150 areas of development 348 00:35:59.685 --> 00:36:00.505 or interesting 349 00:36:01.045 --> 00:36:09.730 new entries to distributed systems design and research that you're keeping an eye on and that you think are going to have a meaningful impact on the industry? In 350 00:36:12.962 --> 00:36:13.462 the 351 00:36:16.935 --> 00:36:17.435 in 352 00:36:17.895 --> 00:36:19.755 the late eighties to early nineties. 353 00:36:20.215 --> 00:36:22.955 We now understand that the state machine approach to, 354 00:36:23.655 --> 00:36:24.155 consensus 355 00:36:25.420 --> 00:36:26.720 systems where you 356 00:36:27.099 --> 00:36:31.040 you run consensus in each operation, you form a log, you execute the operations in order, 357 00:36:31.420 --> 00:36:35.365 that is an effective way to build a state machine. It's how most people use Raft. 358 00:36:35.905 --> 00:36:41.605 That seems well understood. But what's not well understood is how do you do operations between those state machines 359 00:36:41.905 --> 00:36:43.045 and still make them, 360 00:36:43.890 --> 00:36:45.589 legal under the loss of your system. 361 00:36:46.289 --> 00:36:46.789 So 362 00:36:47.250 --> 00:36:48.950 1 approach to that is Calvin, 363 00:36:49.569 --> 00:36:54.835 which is implemented by FaunaDB. And then there's some recent work that I I think Daniel Bhatti has been doing. Is it called the slog, 364 00:36:56.174 --> 00:36:59.234 which looks really cool. Sort of a generalization of this this idea 365 00:37:00.560 --> 00:37:01.060 for 366 00:37:01.520 --> 00:37:13.205 oh, gosh. It's been so long to separate the paper. I'm not even gonna try to explain it. I will butcher it. I'll add a link to the paper in the show notes. But it looks cool. Yeah. Another thing that I'm really excited about is, 367 00:37:13.905 --> 00:37:17.505 revisiting core ideas about consensus. Heidi Howard has been doing, 368 00:37:17.825 --> 00:37:18.805 really cool research 369 00:37:19.240 --> 00:37:38.670 into, you know, do you actually need majority quorums to do consensus? We all thought so, but it turns out that that's an assumption that's not required for the proof in Paxos. And there are generalizations of Paxos where maybe instead of agreeing on 1 value, you agree on a set of values and you can skip around during contention. So I'm excited to see those ideas come into play because 370 00:37:39.210 --> 00:37:43.550 in some cases, it looks like you could do fewer round trips or 371 00:37:43.885 --> 00:37:45.985 shorter round trips, more local round trips, 372 00:37:46.605 --> 00:37:52.305 in exchange maybe for a longer trade off during failover between domains. Another thing that I'm really excited about is CRDTs. 373 00:37:53.230 --> 00:38:00.450 I know I've been plugging this for a decade now, but I think industry hasn't really caught up to this idea of commutative programming. And it is, 374 00:38:01.295 --> 00:38:03.235 at least the Kalb conjecture suggests, 375 00:38:04.335 --> 00:38:05.955 the way that we need to handle, 376 00:38:06.975 --> 00:38:09.155 geographically replicated systems where latency 377 00:38:09.799 --> 00:38:14.220 prevents us from using a consensus mechanism or some other sort of strong consistency. 378 00:38:14.920 --> 00:38:18.059 So I'm excited to see people doing work with causal consistency, 379 00:38:19.244 --> 00:38:19.744 having, 380 00:38:20.365 --> 00:38:26.385 commuted operations which get merged together either via operational transforms or CRDTs or some related structure. 381 00:38:26.685 --> 00:38:28.305 I wanna see more of that. And 382 00:38:28.650 --> 00:38:34.510 you have been running the Jepsen test for a while now. You have come to be seen as 383 00:38:34.970 --> 00:38:36.990 a sort of very strong signal 384 00:38:37.369 --> 00:38:44.815 of correctness in a system or reliability in a system where different database vendors will publish the fact that they have 385 00:38:45.170 --> 00:38:59.065 completed a round of Jepson testing and that they resolved these different bugs that have been identified by it. I'm curious what you see as being the overall impact that your work on Jepsen has had on some of the modern distributed systems products that we're using today. 386 00:38:59.605 --> 00:39:00.700 This is tricky because, 387 00:39:01.140 --> 00:39:04.240 you know, obviously, I have the desire to toot my own horn. 388 00:39:05.260 --> 00:39:12.734 Like, I am the most important person in my own story. But I think realistically speaking, I am a small part of a very large movement to do 389 00:39:13.195 --> 00:39:15.350 more correct distributed systems. 390 00:39:16.210 --> 00:39:17.510 People are advancing 391 00:39:17.890 --> 00:39:21.190 formal methods. Hello Wayne, for

example, has been pushing the use of TLA. 392 00:39:21.810 --> 00:39:24.070 Chris Micklejohn has been working on Coq 393 00:39:24.484 --> 00:39:26.585 and other ways of doing verified systems. 394 00:39:29.445 --> 00:39:36.680 All the people in industry who are writing distributed systems are reading literature and and advancing. And I guess I can maybe claim 395 00:39:37.140 --> 00:39:41.319 some small part in pushing that along, but I think it's a lot of people doing a lot of work too. 396 00:39:41.825 --> 00:39:54.079 You have been working on building Jepsen and evaluating some of these mission critical systems, what have you found to be some of the most interesting or unexpected or challenging lessons that you've learned or outcomes of the work that you've done? 397 00:39:54.540 --> 00:39:55.599 1 thing that was 398 00:39:56.115 --> 00:39:57.495 tricky for me to understand 399 00:39:58.275 --> 00:40:02.695 when I first started this work, and that still surprises people when I teach classes, 400 00:40:03.235 --> 00:40:04.855 is this notion that 401 00:40:05.720 --> 00:40:07.260 you have to record 402 00:40:07.880 --> 00:40:17.964 a concurrent history when you're doing a test. A lot of people are used to doing tests of stateful systems by performing some series of operations in sequence and then asserting the state as a certain way. So 403 00:40:18.265 --> 00:40:22.250 I wanna test the counter. I add 1, 2, 3. I do a read, and the value should be 3. 404 00:40:23.130 --> 00:40:26.670 And that's how a lot of distributed systems tests are are written 405 00:40:27.210 --> 00:40:35.635 at the very first pass, you know, when you're just trying to figure out, does this thing turn on and do its job? But when you're trying to do more sophisticated verification, you have to do these operations concurrently. 406 00:40:36.335 --> 00:40:42.400 And maybe if the read is concurrent with the right, you could either observe it or not observe it. Now there are 2 possible outcomes. 407 00:40:43.420 --> 00:40:47.200 Realizing that I had to track that concurrency and that I had to track indeterminacy 408 00:40:47.900 --> 00:40:49.680 to have a notion that an operation 409 00:40:50.285 --> 00:40:53.744 may or may not have completed and that both things have to be taken into account, 410 00:40:54.125 --> 00:40:55.424 that was a really subtle, 411 00:40:56.924 --> 00:40:59.505 and tricky problem for me to wrap my head around initially. 412 00:41:00.410 --> 00:41:04.510 The other thing that I've really struggled with, and this is more recent, something that I've been 413 00:41:04.810 --> 00:41:06.910 trying to internalize over the last year, 414 00:41:07.290 --> 00:41:07.790 is 415 00:41:08.494 --> 00:41:12.035 what is it exactly that you are proving when you say 416 00:41:12.415 --> 00:41:14.835 such and such an anomaly exists in a system? 417 00:41:15.375 --> 00:41:17.474 Because anomalies, at least in 418 00:41:18.080 --> 00:41:19.780 the academic sense, are 419 00:41:20.800 --> 00:41:21.300 properties 420 00:41:21.600 --> 00:41:24.580 of a formal system. Like like, you have 421 00:41:24.960 --> 00:41:26.020 a set of, 422 00:41:26.645 --> 00:41:41.210 like like, a history is a set of operations, and each operation has the start time and this end time, and and they have some sort of internal order. That's not how real systems work. Real systems are computers. They're they're sand that we tricked into thinking. There are quartz crystals we imprisoned and electrocute in order to watch them dance. 423 00:41:41.750 --> 00:41:48.165 That's you know, there's a mapping between the abstract system and the real computer system you're testing, but that mapping 424 00:41:48.545 --> 00:41:49.045 isn't 425 00:41:49.665 --> 00:41:50.405 always faithful, 426 00:41:51.905 --> 00:41:52.405 nor 427 00:41:52.940 --> 00:41:59.280 is the system you're measuring the actual observation you have. There's a third thing, which is the history you record as the client. 428 00:41:59.795 --> 00:42:00.295 And 429 00:42:00.915 --> 00:42:02.215 so when you say, 430 00:42:02.755 --> 00:42:04.375 I see this particular 431 00:42:04.995 --> 00:42:06.455 anomaly, I see a, 432 00:42:06.995 --> 00:42:09.380 dirty read, or I see a g 1 c 433 00:42:10.339 --> 00:42:12.119 information cycle in a transaction. 434 00:42:12.980 --> 00:42:14.279 What I'm saying is that 435 00:42:16.019 --> 00:42:18.200 were there to be an abstract 436 00:42:18.615 --> 00:42:22.315 system, an abstract execution in this audio formalism, say, 437 00:42:23.015 --> 00:42:28.155 which was executed by this real physical computer system, if those 2 things were truly isomorphic 438 00:42:28.600 --> 00:42:35.500 and if all the information that I got from the system is accurate, if I if I wasn't lied to about the results of any of my rights and reads, 439 00:42:36.025 --> 00:42:40.765 then there cannot possibly have been an execution which did not contain this particular anomaly. 440 00:42:41.145 --> 00:42:42.925 I know this sounds really pedantic, 441 00:42:44.265 --> 00:42:44.760 but 442 00:42:45.400 --> 00:42:49.260 being forced to write it out to develop a formal model of that separation between 443 00:42:49.560 --> 00:42:50.460 abstract model, 444 00:42:51.000 --> 00:42:53.740 whatever the computer did and what you observe of the system 445 00:42:54.225 --> 00:42:58.965 has been critical in the research I've done with Elle to do transactional isolation checking. 446 00:42:59.505 --> 00:43:03.125 And it's guided the way that I talk about anomalies in reports. 447 00:43:03.750 --> 00:43:06.250 I can say something like this appears to events 448 00:43:06.550 --> 00:43:26.859 a g 1 c anomaly, but it could also be a dirty read. And I'm choosing the most charitable interpretation of that fossil behavior. Yeah. It's interesting trying to reason about these systems that have potentially unreasonable behavior because of the fact that physics is a thing, and we have to contend with

that in these sort of idealized systems that we're trying to build and rely on. 449 00:43:27.319 --> 00:43:34.075 Yeah. Physics is a thing. I have a a sort of pet peeve there. This is a bit of a tangent, maybe not related here. But, 450 00:43:34.694 --> 00:43:46.515 people often say, like, oh, relativity means there's no such thing as as synchronized clocks. This is bullshit. Relativity gives us the equations for doing clock correction and and figuring out what time it is. The problem is not, 451 00:43:48.015 --> 00:43:50.595 the fabric of space time. The problem is, 452 00:43:51.214 --> 00:43:54.115 that the clocks are are wonky and generally bad. 453 00:43:54.460 --> 00:43:58.480 And in terms of the work that you're doing on Jepsen and your overall 454 00:43:58.940 --> 00:44:07.385 work with distributed systems, what are some of the things that you have planned for the future or projects that you look forward to being able to start or continue on? 455 00:44:08.005 --> 00:44:08.505 There's 456 00:44:09.204 --> 00:44:13.589 an open problem in my head around l, which is my checker for, 457 00:44:14.369 --> 00:44:21.474 looking at transactions and telling you if they interleaved improperly, if they were non serializable or non repeatable read or violated snapshot isolation. 458 00:44:21.934 --> 00:44:23.474 And that is that I 459 00:44:24.015 --> 00:44:24.674 I understand 460 00:44:25.214 --> 00:44:27.635 the actual formalism for individual registers. 461 00:44:28.410 --> 00:44:32.910 So I have variables x, y, and z. I wanna do transactions over them. That's great. What about 462 00:44:33.210 --> 00:44:34.670 a transaction which says, 463 00:44:35.130 --> 00:44:43.155 read the current value of all registers which have an odd value in them right now? That's a predicate. And predicates 464 00:44:43.460 --> 00:44:49.080 are a part of the actual formalism, which I completely ignored in my initial past because I had no idea how to solve them. 465 00:44:49.460 --> 00:44:51.800 It would be nice if I could 466 00:44:53.425 --> 00:45:04.450 model predicates in history and analyze them in a way which lets me tell if there are anomalies in the same way that I've done for individual registers. I have no idea how to do this thing. It's an open research firm. 467 00:45:04.750 --> 00:45:07.730 And as you continue to work on Jepsen, 468 00:45:08.109 --> 00:45:20.070 I'm curious what you have planned for the future of that project or other projects or research in terms of distributed systems that you're looking forward to either continue or start a new? 469 00:45:20.770 --> 00:45:28.845 The transactional checker I've built called Elle has been remarkably productive in finding bugs. It's uncovered all kinds of behaviors that we couldn't see with earlier, 470 00:45:29.165 --> 00:45:30.545 tests tests in Jepsen. 471 00:45:31.085 --> 00:45:34.545 However, there's a a key problem. We can only identify, 472 00:45:35.920 --> 00:45:38.420 anomalies at the key value level. 473 00:45:39.200 --> 00:45:40.340 So Jepsen's, 474 00:45:41.360 --> 00:45:43.940 or Elle's use of the formalism from Ajah 475 00:45:44.545 --> 00:45:51.445 works over keys and values like x equals 2, y equals 3. You can do transactions over x and y, but you cannot express a transaction 476 00:45:51.745 --> 00:45:52.645 over, say, 477 00:45:53.025 --> 00:45:55.010 all currently odd registers. 478 00:45:55.869 --> 00:45:56.690 That's a predicate. 479 00:45:57.470 --> 00:46:07.225 And odd just formalism includes predicates and talks about anomalies over them. It would be great if I had some sort of first class way to model those predicates and check their correctness, but I don't know how that's gonna happen yet. 480 00:46:07.605 --> 00:46:09.465 That's my my current research project. 481 00:46:10.190 --> 00:46:12.850 That doesn't mean that we can't test predicates in real databases. 482 00:46:13.230 --> 00:46:15.410 You can still make a request to the database 483 00:46:16.030 --> 00:46:28.355 using a predicate in order to perform what looks like a key value read in an abstract sense. And that lets us test whether, say, secondary indexes are correct. But we've got no first class representation of them at the checker level, and I'd really like to get that far. 484 00:46:30.950 --> 00:46:34.970 And for people who want to dig deeper into some of the 485 00:46:35.670 --> 00:46:44.525 transactional guarantees or problems problems in distributed systems, what are some of the resources that you have found valuable or that you tend to recommend to people who are getting into this area? 486 00:46:46.250 --> 00:46:47.950 I've actually written a class, 487 00:46:48.410 --> 00:46:56.994 which I I teach professionally to organizations, but the outline for it is remarkably comprehensive and has lots of links to other resources. So if you search for, 488 00:46:57.695 --> 00:46:58.195 distsys 489 00:46:58.655 --> 00:46:59.395 dash class, 490 00:47:00.095 --> 00:47:03.075 on my GitHub, which is a p h y r, afer, 491 00:47:03.480 --> 00:47:03.980 There's 492 00:47:05.000 --> 00:47:05.660 a whole 493 00:47:06.280 --> 00:47:07.980 introduction to the field. 494 00:47:08.680 --> 00:47:11.900 I also read, like, MiXu's book, m I x u, 495 00:47:12.555 --> 00:47:13.055 and, 496 00:47:13.835 --> 00:47:17.375 Chris Micklechon has a really wonderful reading list for papers and introductions. 497 00:47:17.995 --> 00:47:35.825 Alright. I'll add links to all of those for people who want to dig deeper. And for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as a final question, I would like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today. 498 00:47:36.660 --> 00:47:38.120 Right now, databases 499 00:47:38.660 --> 00:47:39.160 are 500 00:47:39.940 --> 00:47:42.520 often tested via ad hoc tests, 501 00:47:42.820 --> 00:47:44.260 via some sort of, 502 00:47:44.580 --> 00:47:45.080 generative 503 00:47:45.415 --> 00:47:46.234 property based testing like 504 00:47:46.535 --> 00:47:48.395 Jepson or maybe Jepson itself. 505 00:47:49.494 --> 00:47:50.555 But there are 506 00:47:51.175 --> 00:47:54.075 fewer tests of the abstract underpinnings, 507 00:47:54.720 --> 00:47:55.940 either via simulation 508 00:47:56.480 --> 00:47:58.980 or via model checking or a proof system. 509 00:47:59.280 --> 00:48:02.180 I'd like to see model checkers and proof systems become 510 00:48:02.800 --> 00:48:03.619 more tractable 511 00:48:05.505 --> 00:48:12.020 for real industry people, maybe who don't have the advantage of mathematicians sitting at their side to help them write the proof. 512 00:48:12.500 --> 00:48:15.560 I myself struggle a lot to write proofs in Isabelle 513 00:48:16.100 --> 00:48:18.120 or to write models in TLA. 514 00:48:18.500 --> 00:48:23.585 And I would love it if these systems, which think have a lot of promise for industry, were more accessible. 515 00:48:23.885 --> 00:48:45.870 Alright. Well, thank you very much for taking the time today to join me and discuss the work that you've been doing with Jepsen. It's definitely very interesting problem domain and a useful approach that you've built to it. And it's something that I see mentioned very frequently throughout different database platforms and distributed systems. So I appreciate all the time and energy that you've put into that, and I hope you enjoy the rest of your day. Thank you, Tobias. You as well. 516 00:48:51.710 --> 00:48:54.935 Listening. Don't forget to check out our other show, pod cast.init@pythonpodcast.com 517 00:48:57.234 --> 00:49:01.490 to learn about the Python language, its community, and the innovative ways it is being used. 518 00:49:01.890 --> 00:49:03.270 And visit the site at dataengineeringpodcast.com 519 00:49:04.610 --> 00:49:14.055 to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried other projects from the show, then tell us about it. Email hosts at data engineering podcast.com 520 00:49:14.595 --> 00:49:19.815 with your story. And to help other people find the show, please leave a review on Itunes and tell your friends and coworkers.