#163 How to make your models sample faster, with Adrian Seyboldt & Eliot Carlson

speaker-0: My guests today are Adrian Zabolt and Elliot Carlson. I'm sure you know Adrian by now, but this time he's brought Elliot with him, a research analyst in Bob Carpenter's group at the Flatiron Institute, who spent the last year co-writing a paper with Adrian. That paper is about preconditioning, the process of transforming your target distribution into something Hamiltonian Monte Carlo can temple everything. The catch is a chicken and egg problem, since preconditioning requires knowing things about your posterior that sampling is supposed to reveal in the first place. Adrian and Elliott's fix is to use a piece of information samplers already compute, but mostly throw away the gradients of the log density and to formalize the whole problem using something called Fisher Divergence. We get into mass matrix adaptation, why Stan and Pime C's default approach leaves information on the table, when to reach for diagonal versus low rank versus normalizing flow adaptations, and a benchmark results that's hard to ignore. A median, four times speed up across 114 models. This is Learning Bayesian Statistics, episode 163, recorded June 12, 2026. Let me show you how to be a good day and change your predictions after taking information. And if you're thinking they'll be less than amazing, let's adjust those expectations. What's a Bayesian is someone who cares about evidence. Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods, the projects, and the people who make it possible. I'm your host, Alex Andora. You can follow me on Twitter at Alex underscore Andora. like the country for any info about the show. Learnbase stats.com is Lamp Place to be show notes, becoming a corporate sponsor, unlocking Bayesian merch, supporting the show on Patreon, everything is in there. That's LearnbaseSats.com. If you're interested in one-on-one mentorship, online courses, or statistical consulting, feel free to reach out and book a call at topmate.io slash Alex underscore and Dora. See you around, folks, and best Bayesian wishes to you all. Elliot Carlson, welcome to Learning Patient Statistics. And thanks for having me. Yeah, yeah. I'm very happy to have you on. And obviously, Adrienne Sabolt. Willkommen now. Learning patient statistics. ⁓ great to be here. Yeah, always great to have you back, Evyn. you're basically a co host now, so you know you're you're at home. ⁓ feel free to interrupt me whenever you need.
speaker-1: ⁓ and it
speaker-0: And say whatever you want, you know. ⁓ but today ⁓ we're gonna talk about what you guys have been ⁓ working on because you've been working together. ⁓ I think also with ⁓ Bob Carpenter on that project. and so we're gonna talk about a lot of ⁓ algorithm nuts, nut by ⁓ Rust, th things like that today. ⁓ it's gonna be a a fun episode. But to start with, of course, Elliot, it's your first time on the show, so ⁓ we need your reaching story. Okay. What what are you doing nowadays and how did you end up doing that? Okay, well I'm currently a ⁓ research analyst in Bob Carpenter's group at the Flatiron and also I've been the past year working with PyMC Labs, to whom I was introduced via Adrian. ⁓ so I did a undergrad in in stats and a masters in stats and sort of I think I I arrived at Bayesian stats the reverse way that most people do, where ⁓ or ⁓ excuse me, to ⁓ DMC in the reverse way. So I first learned about MCMC without really knowing much about ⁓ Bayesian stats and was interested in in that. And ⁓ so I ⁓ was introduced to Bob and asked if I could ⁓ if he had a project, an MCMC related project for me to work on. And then he ⁓ introduced me to Adrian. He said I have a an algorithm I've been trying to understand for a year now and but I haven't had the time to fully dive into it. So then that that initiated maybe a another a year long process of us writing this paper, which was a lot of fun and and you know learned ton.
speaker-1: This Adrian guy just never wrote down his algorithm in a proper way, right?
speaker-0: I mean, I understand you, Adrian. Like writing papers is is not the funniest part, I would say, of of the job. ⁓ it's not the one I enjoy most. So, you know. ⁓ I think it's much, much more fun when you do it with other ⁓ interesting
speaker-1: Actually writing it with ⁓ with Bob and Ali it helped so much. That was a lot of fun. So
speaker-0: Yeah, yeah, exactly. Writing alone is just ⁓ my god. Yeah, I hate that too. and so Elliot actually so you you answered that a bit, but ⁓ I'm really curious also to what drew you to patient inference specifically. ⁓ because I think here you you told us how you end up ended up working on HMC basically and on samplers. ⁓ is there w were you introduced to patient stats before that or Did you discover them also just with this ⁓ with these sample with this sampling work? Yeah, so I think the chronology was I was first taking a class in stochastic processes and Markov chains, and then I learned about Metropolis Hastings, and I thought, ⁓ wow, this is I don't know why, I just thought, wow, this is really cool. ⁓ even though I I didn't really know what it was used for at all. ⁓ which is kind of what I meant in like reverse way, because I think most people view sampling as like this ugly thing they have to do when they want their models to fit. But anyway, so once I you know, in the past couple years, now that I've been exposed to more you know, of the Bayesian methodology, I I become more interested in that in and of itself, I think. ⁓ especially just seeing how powerful it is for especially like researchers in in you know, scientific domains who have prior information from other exper experiments and things like, you know, Adrian and I have worked on a project where this has come up a lot. ⁓ where yeah, people have priors or, you know, from other experiments that they need to integrate into their models. ⁓ and then but I I would say that Adrian may be the same, though the fun comes in the sampling part. Yeah. I think ⁓ think Adri Adrian will relate to that. And ⁓ well actually Adrian, it's your third time on the show, so I will refer to ⁓ to your previous appearances, ⁓ episode, episode seventy four, I think, with your first one where we talked about your well that was, you know, I think the first time. Yeah. and and that was the first time we talked about Zero Subnormal and also about Nutpie and and the work you did on that and why you started doing that and why Rust is interesting. So I encourage people to listen to that. ⁓ episode seventy six also with Bob Carpenter, ⁓ which is a great background. And episode one thirty three, ⁓ where you joined as a guest star with ⁓ Sean Pickney, ⁓ because we talked about all things ⁓ algorithm efficiency but also Zero Subnormal in Stan, ⁓ which ⁓ Sean implemented so that was a nice crossover episode. ⁓ so I recommend listeners to to to listen to these ones but ⁓ maybe can you tell us, you know, ⁓ since then what's what changed or what evolved in the project, especially NetPy and also what you've been working on lately beyond the other paper, of course.
speaker-1: Yeah. So kind of in NutPie itself, kind of from a high level, I would say things haven't changed that much. So the basic mass matrix adaptation algorithm has stayed the same. There was a little bit of tweaking, like the the window sizes are different and they are growing now, things like that. Kind of around that though, the project has changed quite a bit, I would say. So got a lot more robust, so it has a lot more users, so they're definitely notice whenever some whenever whenever I break something, people actually report reported in issues, which is ⁓ a strange feeling. So you you notice that people seem to be using it now. so yeah, just has been a lot more tested, I would say. I guess one bigger ⁓ new item is that it has an R interface now as well. So not just a Python interface, but an R interface. That's actually not written by me. ⁓ somebody else wrote that and was really happy that that appeared. So ⁓ People can now use it from R as well if you have stand models or something like that. And I think that's probably the the biggest change there. Also on the mass matrix adaptation side, the load rank version, which has been around for a long, long time, it it works now much better. So when I tested that years ago before writing the paper, it was like somehow it didn't really work that well. Then when we wrote the paper, I just ran the benchmarks again and suddenly it was working well. I never was able to figure out what what really changed. Maybe my benchmarks were just wrong or I I no idea. I had the idea it should work well, but it somehow didn't for some reason. Now it does. So ⁓ I'm not sure if that's actually a change in NutPie or just ⁓ me looking again. but yeah, that's ⁓ definitely worth a look if ⁓ to for anybody who's working on on models that are not trivial to sample.
speaker-0: Yeah, yeah, actually we can we can talk a bit more about that and I think it's a nice segue into your paper. ⁓ so can you can you tell people what even you know mass matrix adaptation means? ⁓ I guess it's ⁓ it's a complicated word, ⁓ an intimidating word when you haven't heard about that. So what even is that about? Why do you why would you even care about it for a for a model?
speaker-1: So they're diff completely different ways of thinking about the mass matrix. One way is it defines an inner product and you need an inner product for nuts. That's kind of the path side. If you're a mathematician, that maybe tells you something. Otherwise, maybe that's not too helpful. Kind of if you have divergences or something, the sampler always tells you, yeah, you you need to re-parameterize. So you need to find new parameters, new way to write your original model. And The way I tend to think about the mass matrix adaptation most is as an automatic reparameterization in a sense. Just a really, really dumb one that doesn't fix much, but it fixes really, really important things. Namely kind of the diagonal version of mass matrix adaptation just rescales your variables.
speaker-0: I would also add that like yeah, I I think the sometimes the term mass matrix it sort of drags you into the physical interpretations of HMC, which are really beautiful if you understand physics, which I do not. So but but really like you can just think of it as ⁓ what we call like preconditioning, so just changing the geometry of your distribution, and you can show that. Like using a specific mass matrix is the same thing as just sampling from this ⁓ sampling from a slightly, you know, shrunk or stretched ⁓ distribution or or a distribution with that's some linear transformation has been done that transformation be in the mass matrix. Yeah, thanks, Eliot. I th I think that's a very intuitive way for for people to understand it. And actually, so each time each time people use Pine C or Stan or other PPL, there are doing that kind of things under the hood, ⁓ even though they may not know. And and that's the that's exactly the beauty of PPS, right? It's that you don't have to care about that. ⁓ And so yeah, HMC has that large, if I understood correctly and you guys reframe me if I'm wrong because you're the expert, but it's it's a ha it has this like large constant factor that's determined by the curvature of the target density. And so what mass matrix adaptation does, as you said earlier, these preconditioning on that to make sure we're able to sample from the posterior afterwards after we've tuned. but yeah, how does that work currently ⁓ in Stan and PyMC for instance? And what does NutPy do differently so that we understand a bit a bit more the difference and the The value proposition if you want of NutPy and then we'll dive into your newest paper.
speaker-1: So ⁓ the l let's stick with the diagonal mass matrix for a second. ⁓ so that you can think of it looks at each parameter and tries to just rescale it. So if it's maybe has a posterior standard deviation of ten to the minus four, then it will just scale it up by ten to the four ⁓ to make it bigger. So that kind of the the the the posterior standard deviation of each parameter as the sampler sees it is identical. And that's exactly what Stan and kind of the default PyMC sampler, what they are doing. They're just saying, I look at the posterior standard deviation and I rescale it such that the posterior standard deviation is one. And I mean you could do that manually if you really, really wanted, just introduce new parameters that are rescaled versions of the original ones. You just wouldn't want to. That's not a good use of your time. So we do it in the sampler. ⁓ So Stan and PyMC, they just use the posterior standard deviation directly. We look at the second source of information, we look at the at the scores, so the the d ⁓ the derivatives of the lock ⁓ lock density. Kind of in an ideal world you want that the standard deviation of your parameter is one, but also that the standard deviation of your scores, so of those derivatives, that those are also one. They are both kind of interacting in in in in some sense. And NutPy just uses the geometric mean of those two standard deviations. ⁓ and then there's a long discussions kind of how you can can kind of say why that is a good idea. That's what we go to in the paper a bit more, and kind of a much more general framework of kind of how to do good preconditioning. And kind of this this particular version just pops out of that as a special case. But in this special case, it's just the geometric mean of those two. two things that we kind of want to want to rescale in a in a sensible way. And usually so that's there's no guarantee there you can if you if you work on it, you can find examples where it's worse. but usually it's better. And ⁓ usually is I think where samples are concerned, the the best you can pretty much always that's the best what what what you can do. So ⁓ yeah. ⁓
speaker-0: Eliot first, ⁓ maybe you have you have something to to add or ⁓ yeah, anything you're thinking about on that? ⁓ I would say maybe this is ⁓ jumping ahead, but ⁓ you know, one ⁓ basic basically w you know a lot of what the paper is doing is formalizing some intuitions and general things that we want out of the sampler. So like one ⁓ one thing I would add is that we we basically know that ⁓ standard normal is ⁓ very perform HMC will perform really well on a standard normal. And you have this nice fact about standard normals relating the ⁓ covariance to the ⁓ the covariance of the draws of the distribution to the covariance of its scores. So if you think about ⁓ so if you have like a you know a normal distribution is ⁓ parameterized by a mean and covariance. And so if you look at the if you look at the distribution of the scores, the gradients of the log density, their covariance is the inverse ⁓ sigma, the inverse covariance of your normal, which is kind of this cool fact. And so that provides a little bit of intuition as to why. So so then when we take the ⁓ metric mean, we sort of want to combine those two pieces of information. ⁓ so we'd look at the inverse scores and the the draws. Okay, yeah. Yeah. ⁓ actually I think that's a good that's a good moment to ⁓ to dive into your ⁓ your paper which is in the show notes for people who wanna dig deeper because here of course well ⁓ we won't make it justice, ⁓ because it's ⁓ it's a very ⁓ thorough and and deep paper. ⁓ so titled is Preconditioning Hamiltonian Monte Carlo. I think all of that we just talked about, by minimizing fissure divergence. So We'll go into that second part, ⁓ just after what's. But first, can you maybe give us the elevator pitch for the paper and and tell us what what it does intuitively? So the big problem that Stan faces or any s HMC sampler is that when we're you know, preconditioning, basically we want to find a some mass matrix that ⁓ turns our distribution into something that's friendly for HMC to run on. And to do that, you need to know something about your distribution. You want to know maybe it's covariant structure or something. But the whole point of doing sampling is that we want to learn about our distribution. So, you know, things like its covariance structure. So that we call that this this codependence, ⁓ the chicken and egg problem. It's known as. So so basically, ⁓ you know, ordinarily like preconditioning based on draws is how ⁓ it it we sort of initially come up with like a very rough sketch of what the distribution might be based on just a few warm-up draws and then and then sort of get the motor going. ⁓ and so that can often take like a huge amount of the sampling time. And so basically the so as we said, the, you know, we have this other piece of information in the form of ⁓ scores that samplers like Stan are not using. And ⁓ that can be like really valuable for preconditioning and since we have them we might as well try to use them. And so a lot of the work of the paper is formalizing ⁓ you know, what information can these scores give us about ⁓ the the distributions and so and formalizing like how how do we turn this into a ⁓ how do we best make use of this information, I guess. Yeah. ⁓ yeah I think that's a great explanation. Makes it a bit more intuitive for people. ⁓ Adrian, do you wanna add anything to that?
speaker-1: Yeah, I guess what I would add is the idea of the general framework of how to do preconditioning. So kind of what the paper really does spend most of its time on is a special case of this. So we say we have this objective how we want to ⁓ precondition ⁓ the model. And mostly the appendix kind of have such in a very general way written down. ⁓ and then most of the work is What does that do in a very special case where we just look at the mass matrix that does linear things? And there we work through kind of what has that, what what are the in influences on the condition number, why does this work better kind of but but as I said, just in the linear case. And what I really like about this framework is that it's not restricted to that. You can add nonlinear transformations as well. ⁓ that's kind of where the normalizing flow adaptation that's still experimental in in NotPy, for instance, is coming in. and I think that's where a lot of the interesting research, more researchy side of things look like. Kind of the the linear one I would more say is more in the engineering part where okay, we have the objective, we derive it, we look at look at derive closed form solutions, we look at the the details ⁓ of that, the closed form solutions and inspect what it does in that particular simple case. And on the research end, we kind of have this. How can we generalize this? And what's that what does this tell us in general about how to precondition ⁓ HMC?
speaker-0: Actually maybe we can talk a bit about the normalizing flow adaptation because I think it's a pretty intuitive concept and that will that will set us up well to then talk a bit more about feature divergence and and mass matrix adaptation. So yeah, can you or Elliot give us the the idea of why normalizing flow adaptation would be interesting here? and what it does. And so you don't need to redefine what normalizing flows are. I have an episode on that with Marilou Gabrier. I'm gonna put it in the related episodes. But yeah, feel free to just like give the elevator pitch for it and then make it in the context like give us in the context of NutPy and Mass Matrix adaptation what it does and why it can be interesting. I think that especially in applied problems, there are a lot of ⁓ geometries that are induced by certain parameters that create ⁓ that that basically are not solvable with a linear preconditioner at all. So the most famous example of this is like the funnel, which arises when you have these hierarchical models where there's a variance parameter. So you get this, you know, big thing that's called a funnel. And basically on either side, any any linear ⁓ preconditioner that you try to apply to sort of ⁓ tame one side of it, you know, this very thin over here and very wide over here. Anything you do to one side will just make the other side worse. ⁓ so ⁓ normalizing flows since they can sort of ⁓ since it's a a neural network, it the you know obviously very not restricted to be linear. ⁓ and so you can get kind of ⁓ sophisticated, more sophisticated preconditioners. And yeah, as you mentioned, nor normalizing flows, you have the ⁓ they're designed in such a way that you have the normalizing you have the Jacobian, excuse me. ⁓ and so that that can help you. Yeah. So I actually I have not used normalizing flows myself, but I am I would be interested in in trying them out. Yeah. Yeah. Yeah I've tried them I tried the adaptation ⁓ of normalizing flows. ⁓ but it was in a huge GP model and in the end it didn't like it wasn't giving ⁓ speed ups for assembly. Actually the the classic mass matrix adaptation was perfectly fine. ⁓ so it really depends on the models and ⁓ on the geometry ⁓ that you get with the the posterior and if it's especially complicated with well, you know, funnels as you were saying, then it's gonna help a lot because the what I find super Intuitive and interesting with the normalizing flow idea is that basically it just ⁓ tries with a neural network to learn a transformation of the space that becomes a standard normal. and that really ties to what you were talking about at the beginning of the episode where, well, you know, standard normals are super easy to sample in HMC, so why don't we try to do that? And Yeah, the idea is that if you have a very complicated posterior geometry, spending more time in tuning and adapting the mass matrix with the normalizing flow can be very interesting because then it's gonna pay off a lot afterwards when you start sampling. ⁓ obviously the caveat is that, well, you have to learn a neural network, so ⁓ tuning's gonna take much more time. But basically the idea is a bit like to amortize that. ⁓ that tuning time with then really fast and efficient sampling. ⁓ and that's also why like pairing that adaptation with normalizing flow with GPUs is gonna play extremely well, ⁓ especially if you are using NutPint to do that. So yeah, like it's it's interesting to have in your toolbox for sure, even though you're not gonna use that all the time and really not the majority of the time actually. This kind of I would say like ⁓ leads into the the Fisher divergence topic. I don't know if you want to ⁓ talk about that. Basically Yeah that's perfect. Exactly what I was gonna ask you. Yeah. ⁓ so just I'll give I'll give like a brief a a simple background, I guess. So most people know about K L divergence. It's basically just divergence as being like a way of trying to compare ⁓ I don't I mean I'm gonna use the word metric. I know they're not like technically in metrics, I think, but but basically a metric of comparing, you know, how close are two probability distributions to one another. So the KL divergence is the most common one where you you know you're integrating over the the support and you know comparing the the log densities. The Fisher divergence is the slightly different ⁓ metric which where you're comparing the scores and you're basically saying, okay, you know, distribution A and distribution B. ⁓ when we integrate over those, how close are the scores to one another? And so the so we can use that as a ⁓ minimization objective for preconditioning. So how do we get our our ⁓ target distribution how do we minimize the fisher divergence between our target distribution and a standard normal, which we know works really well. And so that's Yeah, that's all that Fisher divergence is sort of a formalization. And and the nice thing about that is that the the solution to that minimization problem incorporates both draws and scores, which are exactly the two pieces of information that we have ⁓ during preconditioning. So it sort of ⁓ works out works out really nicely. Interesting. So what is the two two follow up questions? what is the difference with KL? divergence with most of the listeners will know about. And why is Fisher divergence here a better objective ⁓ for what we're talking about? And maybe additional question ⁓ completely random. Do you know why it's called Fisher Divergence? Which is ⁓ a like a very difficult, like non intuitive name? I actually don't. Adrian, do you know why? ⁓
speaker-1: I guess ⁓ because of Fisher, kind of he was doing a lot of work with sc with the score function, so I would assume that's related. It's also closely related to the Fisher information matrix or Fisher information ⁓ in general. So I would assume it's coming from that, but I never really checked. It's also really pretty much the same thing as score matching, kind of in different literature, that's ⁓ it's also called called that. okay.
speaker-0: Nice. Well after that ⁓ deraning of the of the thought process ⁓ because of me. So what is the difference actually with with KL divergence?
speaker-1: so the KL KL divergence really looks at the density and the Fisher divergence looks at the scores, so the derivative of the density. And it turns out that you can use both kind of if you know the the full if you know know if you have a cop probability distribution and you know what the derivative looks like everywhere, then you know the full ⁓ the the full ⁓ the full distribution. Similarly to how if you know the Density, you the full distribution in a sense. So that those are really complementary. You can use either one or the other one. ⁓ the reason the Fisher divergence in particular or the two reasons the Fisher divergence is ⁓ particularly useful for us. One is that it just uses the information that we have, which is a nice thing, because we don't really have the density in the first place. We have only have the density up to a constant, but we do have the derivative, the score function directly, because the constant goes away. when we take the derivative. So that's nice. the other reason is more related to HMC ⁓ directly. So ⁓ HMC really uses the score function to drive you kind of to to find the trajectory through the parameter space. So that's really the thing that we want to target in a sense. So we want the trajectory of ⁓ our sampler to look like the trajectory of a standard normal distribution in a sense. And that's because that's what makes it efficient, the the trajectory. It's not so much the density in itself. ⁓ so for HMC specifically, I think it's the right thing to target. You could actually add additional terms, but that's a different question. So you could look at higher order derivatives as well. I think that's res d interesting research ⁓ direction as well.
speaker-0: Yeah. Super thanks, ⁓ Adrian. Super clear answer. So and that ties well with ⁓ ties up well with the, you know, physics interpretation that you were talking about before, Al Yad, because like, yeah, okay, the the deriv derivatives, like the second order and ⁓ like kind of the ⁓ if I remember correctly, if it were a car that'd be the the speed, right? Or that'd be the acceleration. I I I always mix mix them up. ⁓ but yeah, like Okay, so it's like you're interested in the second order the change in the derivative and not necessarily and not the in the density, sorry, and not the density itself. And that's what the Fisher divergence tell you. ⁓ divergence from what though? Like it's so it's like you look at the density you're trying to sample, ⁓ the the direc derivative of the density you're trying to sample, and you're looking you're comparing E to something to say that it's diverging?
speaker-1: So a divergence is just the math term to kind of like a distance like thing for different probability distributions. It's not really a distance because it's not symmetric, for instance, but it's like a math term for distance for probability distributions. I'm actually also not entirely sure where that nomenclature, where that name is coming from. But ⁓ so it's the yeah, how different is our standard normal to our transformed posterior? That's really what that divergence isn't measuring there.
speaker-0: Is that Yeah. Yeah, and a and also if I remember correctly, it's like we're talking about divergence and not distance because it can be asymmetric. and it it doesn't have to be symmetric. So so that's one of the of the main thing. I think earlier to you wanted also to to answer my my question. Yes. W ⁓ wait, which which question? Sorry, I've I forgot. Divergence to what? And I think what you were saying is that well we are looking at the difference between like the the distance again in in quotes, so but the divergence between the target density, well the derivative of the target density and I'm guessing the derivative of a standard domal, I think was your answer. Yeah. Yeah. So I think like underneath the hood maybe it's a little in the weeds, but ⁓ like at a given point you sort of want to you know, you have the of your target, you have the point and the score at that. ⁓ at that point and then you'd want to say okay well if this were a standard normal what would what what would the score be at this point and then we want to change the distributions that those scores match. Yeah. Okay. Yeah, yeah, yeah. ⁓ that's that's very clear. And now actually I'm so I'm wondering you in the paper you present three version of these of these adaptations and we've also mentioned a few of them here already so it's like diagonal, dance and low rank plus diagonal mass matrix adaptation. So can you walk us through those and when you would reach for each one?
speaker-1: So the diagonal one is just the default. That's kind of the simple one. That's really, really fast. So it doesn't really add any overhead. So if you've got a really nice posterior, no correlations or something, the diagonal is the one because kind of that does the work, doesn't spend much time doing anything else. the dense one is really just a mass math thing that we needed in the paper. So that's not even implemented. You could implement it. There's nothing particularly wrong with it, but I think the low rank plus dense is ⁓ low rank plus diagonal is really kind of the practical thing that that that you want if you want if you want to have ⁓ a way to fix correlations in your posterior. So the dense one is just we need that to derive the equation for the third one. and the third one, the diagonal plus low rank, really is. Rescale it first with the diagonal mass matrix and then ⁓ try to fix some correlations that you can find. And in a way that it scales reasonably well with the dimensions. So kind of if you were to look at the full mass matrix, it would get really, really slow because that's just computationally a lot to do. ⁓ in unless you have a very, very low-dimensional problem. So we use a low rank version of that. It still adds some computational overhead, but it's not not crazy, so it scales reasonably well. ⁓ it does usually add a couple of seconds or so, maybe to the total sampling time, maybe a bit more. So if you're really fast that you don't want that. If if you have a larger problem, it doesn't matter at all. It's like the that that's a rounding arrow. And I've actually thought about making the low rank plus ⁓ the low rank version the the default at some point. That's not the case yet though. But you can think of it, I think, by now as a reasonable thing to do and and not just this crazy now I need need the the big guns thing. Like that's that that's the flow the the the flow version. That's kind of the crazy one. But the low rank version I think that's mostly ⁓ mostly quite workable by now. You might end up in kind of certain situations where something's unstable or so it might not work even though the diagonal one works if something's ill conditioned somewhere. So that can happen, but I don't think it's particularly common.
speaker-0: Okay. And and so yeah, actually maybe the normalizing flow adaptation would be the fourth one, right? The the last one you reach for once you've reached for the the three previous ones and and they are not working for your case.
speaker-1: Yeah. So the the flow version I'm mostly a bit skeptical of kind of telling people to use it because the implementation is a bit messy right now. So you might get weird error messages. It's slower than it could be. so there's a lot kind of to to do on the implementation there. Math wise, I'm relatively happy with it so far. ⁓ but just the engineering takes ⁓ I think takes a bit more time. Also kind of kind of saying a normalizing flow is doing a lot of work there. Kind of what normalizing flow. There's like a bazillion different ways of doing this precisely. And which one we kind of would want to be the default is like I don't know how to choose that at this point. so that is a big question. Like what particular normalizing flow? And we have one that's going to change for sure. So if you have a model that works now, I can no not guarantee that it will also work in a future version if we happen to change the ⁓ the normalizing flow for instance. Then that will definitely break some things that worked before and probably hopefully fix ⁓ more things that yeah. It's like that if you change those ⁓ those settings. ⁓ so that that I would think of an experimental thing that I'm happy if people try it, but do expect a couple of round ⁓ rough ⁓ yeah, roughly
speaker-0: Fetches, yeah, for sure, for sure. ⁓ and actually so I've I have a question related to that. But first, Elliot, do you wanna add anything about this ⁓ presentation of the different method of mass matrix adaptations? Yeah, well, I w I was gonna add ⁓ anecdotally, like I I mean I pretty recently had a GP model that w took like an hour to sample with ⁓ diagonal and then changing it to low rank it was like a matter of seconds. Like the speed up was was crazy. ⁓ so wow I would definitely like you know if it it's definitely worth trying out. I especially for for I think models where you have some reason to think that you might have like a very correlated posterior for some reason, either due to like constraints or I I I didn't go under the hood and really figure out why it was making that GP model so much quicker, but I think ⁓ it could have something to do with that it was a a GP too, maybe it works particularly well for those, I'm not sure. ⁓ but yeah, so I I would encourage people to try that out if they're ⁓ if ⁓ if it's taking a long time with the diagonal, it could it could spe ⁓ give you an insane speed up. Yeah. Yeah, so you mean ⁓ going from the default diagonal adaptation to the ⁓ low rank plus diagonal one, right? Interesting. Was it ⁓ was it a vanilla GP or an GP approximation like S A C S G P or something else? Actually don't remember. It might have been vanilla. ⁓ yeah, okay, yeah. Yeah, that's my intuition too. Again, my experience it's gonna it's gonna help with Penny L IGP, but if you're using an approximation already like HSGP, which was my case actually for the model I mentioned earlier, it's not it's not helping much because the adaptation, like the diagonal adaptation is already working well because the model like the geometry is much easier because you thanks to HSGP you have it's much more of a l almost linear problem. ⁓ and so that makes for a a much easier geometry. Which leads to actually my follow up question on that, ⁓ maybe for Adrian first and then Elliot, it's okay, so we have these different ways of doing the adaptations. But the issue is that and I guess it's gonna be for most users, you know, like not not for you because you have an in an intuition and once you start working on these models you get an intuition of what's what's gonna work best. But my question is, well, if I only know what the best adaptation is once I've sampled my posterior model It's not that useful, right? ⁓ and that makes the the iteration much longer than it needs to. So is there a way and even is the research and something you guys maybe are working on, which would be like ideally for the users, I'm thinking, well, we do that ad adaptation ourselves, you know, under the hood, and we're able to more or less understand what the posterior geometry is gonna look like and then we just under the hood, not pain's like, ⁓ here I'm gonna use low rank plus diagonal because it's gonna work better without even the user no knowing it instead of doing it manually and incrementally. ⁓ so yeah, like is that something possible and what would you tell people right now to to do in the
speaker-1: I it it's a good question for sure. Kind of the this idea of users just trying lots and lots of different sampler configurations is not something I'm actually too happy about. So kind of this stand way of here is the sampler, write the model, if it doesn't work, fix your model. I think that's in many ways that's the right thing to right way to think about if you're use building statistical models and working with them. If you start playing with the sampler and spending lots and lots of time of playing with the sampler, at some point it's it's just ⁓ much better to fix the model. Just fix the model and the sampler will not be the problem anymore. As a developer, that's of course different because I want to kind of increase the size of things that that work out of the box, right? So there's a bit of a same. Yeah. ⁓ So automatically choosing. I think that's what we want.
speaker-0: Heh.
speaker-1: And in a sense, that's what the low rank version already does. So it will kind of compute the full or not the full, it will compute a mass matrix with a larger number of eigenvalues and then throw away those that look like they are too small to do anything useful. So it only keeps those eigenvalues that look like they do useful work and it gets rid of the other ones. ⁓ so there's still the additional cost of figuring out what were would would have been the options, right? There's also the cost of okay, that there's this additional cost, so we don't want to do it every step, for instance. Kind of the the diagonal mass matrix we update in every draw. The low rank one we don't update in every draw because that's takes time. So there there is a trade off and I don't don't know way of getting rid of it entirely. But as I said, we are trying to kind of make the low rank version kind of the the thing that chooses automatically. But how well we are there, that's yeah.
speaker-0: Yeah, yeah, yeah. No, I I I understand. But yeah, that'd be a big rate. So and it sounds like at some point you're gonna make the low rank adaptation the default in NetPy once you're happy with ⁓ what you're working on right now.
speaker-1: Mm that might happen, yeah. ⁓ I don't want to kind of set that say that promise that in any way, but ⁓ I think that might happen.
speaker-0: Mm-hmm. Okay. Elliot? Anything you want to add? Yeah, I'm ⁓ I I would add I guess that yeah, I think for at least in my experience, I hope I think Adrian would agree. Like if for any sampler you can always conceive of a posterior that will, you know, will totally thwart it. Yeah. ⁓ you know, you can always have either like tail behavior that has some totally different covariance structure for some reason or whatever. There's also just the classic, you know, there could be a mode a million miles away that you'll never find. So I think it's a matter of as Adrian said before, of what works on average better. Yeah. ⁓ and then yeah, I I would add to what Adrian I I I feel the same way in that there's sort of an interesting ⁓ kind of tension between Like you want, you know, the the software to be as user friendly as possible and and but you know, you also don't want i if if a some posterior is really horrible and ⁓ the user just presses go and you know, you you'd sort of like there to be some feedback that, okay, you know, we we can sample this, but you should know that, you know, we you know, we had to whip out this normalizing flow to do it. So you should also maybe consider, you know, reparameterizing or something. I know also there's some work ⁓ on automatic like reparameterization before like even pre pre not in the sense of how we've been talking about it, but pre sampling, you know, via a pie tensor or something if if you know they see see some pattern in in a model to ⁓ just rewrite it, which I I don't know too much about, but I I I believe that's that's going on, which would help to Mm-hmm. Yeah, yeah, yeah. ⁓ that definitely is going on. I don't know if you're working on any of that, Adrian. ⁓ if you are, we can we can touch on that topic ⁓ a bit later in the in the recording. But ⁓ I think actually it's ⁓ it's ⁓ good time to also talk a bit about the the benchmarking that you're doing in the paper, ⁓ because you're actually doing exactly that kind of benchmarking with ⁓ The lowering plus diagonal mass matrix adaptation, in in the paper where you're you're ⁓ comparing a lot of models like ha like if I remember closing more than one hundred models and you're showing that the lowering plus diagonal approach outperforms the the industry's standout by a big factor. Don't remember the exact one, but ⁓ it's ⁓
speaker-1: It's medi the the median difference is like a factor of four. So it's ⁓ sometimes it's much, much more than four, but ⁓
speaker-0: Right. Yeah. Yeah. So were you surprised by by how big ⁓ the gap was? And do you have any do you have any idea of why that adaptation works ⁓ so much better? And also well addendum ⁓ to that question is you were saying that this adaptation is more or less the ⁓ case where NutPy is trying to automatically guess ⁓ the difficulty of the posterior. So I'm I'm guessing this is this is this is related to that. But yeah, like can you explain also the intuition behind why you're saying ⁓ that
speaker-1: Yeah. So what the low rank one does fix that the diagonal one doesn't fix is correlations. So static correlations. So you can have correlations that change depending on some other parameter. It won't will not fix that. It will also not fix funnels or whatever other kind of weird crazy geometry you have. But if you have correlated posteriors, it can fix those. Up to a point, there's also kind of if especially if you have a very, very large model with lots and lots of parameters, then kind of estimating what the correlation is gets harder and harder. So it will fix start fixing things less. but still correlations just are very common in models. We know that for various reasons. For instance, if you use if you have hierarchical linear regressions and you use centered parameterizations, which is kind of usually what I write down first And you have lots of data, you end up with strong correlations. sorry, if you use non-centered parameterizations, then you often end up with very strong correlations. Or if you have over parameterizations, you end up with often very strong correlations. So there are lots of reasons why many models will have large constant correlations in them. And fixing those it it makes sense that fixing those helps a lot. so in in that that regard it's not not surprising. Then the question is kind of how common is that actually in models? And that kind of goes down, get gets back to kind of how difficult it is to actually benchmark samplers. Because kind of if you look at papers, it's quite common that papers will cite, here are five models. I tried it on those five models and hey, if it works, yay. But I mean you don't even need to to be maliciously cherry picking for that to be a problem because I mean maybe you developed your sampler with those problems in mind and you chose your hyperparameters such as works there. That happens incredibly easily. So you need more models to really evaluate it, but where do you get them? Where do you get kind of what in what what fields different fields use completely different models, have completely different typical posteriors. What do you use there? So the best I found the best set of pot models that we have is posterior D B, I would say, and it's it has big gaps. It doesn't have that many really big models. It has kind of lots and lots of relatively small models that are sometimes very easy, sometimes very hard. And it also has larger models. It's not not like there are none in there. But ⁓ so you all still need I mean it's a bigger benchmark I would say than most other papers have, but you still need to take it with a grain of salt because like your models might be different than the models that happen to be in posterior DP. But in general I would say yes, it makes sense that ⁓ the low rank version works better in many actual models.
speaker-0: Mm-hmm. Yeah. I think that's really ⁓ a takeaway for most of the listeners here, right? It's like first, ⁓ honestly if you're if you're using Pine C models, just use the NutPySampler, honestly, by default. ⁓ that's that's what I do in in my own workflow and and you're gonna get like even out of the box, you're gonna get a really sizable speed up on on CPU. And then if you're using GPU even more, ⁓ and you have all these different mass matrix adaptations, and then well definitely try the ⁓ the Dag Diagonal Plus Low Rank ⁓ mass matrix adaptation. I would say even like almost out of the box. That's I'll definitely do that now. ⁓ each time I have ⁓ I have NutPie on my on my environment. So ⁓ yeah, that's a great takeaway. ⁓ Elliott. I know you worked on that too, so I'm guessing you have ⁓ very interesting thoughts also about ⁓ what we just talked about. Yeah. I would I I think I would also add ⁓ that I I believe posterior d B is open, you know, so if if you, you know, have a model that isn't running well and or or is sampling well, I think you can always add it to Posterior D B. And I think there's a lot of, you know, ⁓ more models it it would be great. to have more models ⁓ in there to reflect ⁓ like the diversity of of models that are being used in different in field different fields and whatnot. ⁓ so I think that's very much like a you know an ongoing thing. I think there's also one in ⁓ in blackjacks they have inference gym as well, ⁓ which I think has some very different models from what I if if I remember correctly. ⁓ Yeah, and and I know there are some people working on on you know, building those up so that they reflect, you know, kind of equivalent to what they have in machine learning with all of these, you know, huge benchmark data sets be great, you know, to keep adding new posteriors in. Mm-hmm. Yeah, and actually can you we We haven't talked about Poster T B mo mostly on the show, so ⁓ I think a lot of users, ⁓ listeners don't know what we're talking about. So can you actually define what what this is, Elliot and
speaker-1: It's basically just a repository with a couple of like a hundred and twenty or s something like that. Mostly Stan models. Yeah. I think a couple of them are ported to PyMC. More are ported in a pull request somewhere, but it's ⁓ most of the models as I said are in in in Stan. with the corresponding data sets. So you can kind of ⁓ mix those up and and and and run those. So not nothing particularly fancy, but really, really valuable.
speaker-0: ⁓ okay. Yeah. Yeah, it's a it's a great effort to try and benchmark the models, basically. ⁓ the not the models, the samplers. ⁓ and as you were saying, it's a very complicated job to make an apples to apples comparison and basically this ⁓ this effort is trying to to make that try, to do that right. ⁓ and actually I'll ⁓ put a link to Poster TV in the in the show notes. I think it's ⁓ interesting for for listeners. and Well, Adrian and Elliot, you you both mentioned about that low rank plus diagonal approach. So it works incredibly well in a lot of cases, but also on some problems it doesn't work at all. And it can fail spectacularly. But you don't have a clear idea and understanding of why right now. but what are your hypotheses? You know, what are what are you thinking? Their problems might be. Elliot, maybe you can start and then Adrian? Sure. I I think, you know, ⁓ as I said before, like it's always going to be an issue of averages and and so sometimes ⁓ for specific models, you know, you're always initializing somewhere that can be completely arbitrary. And, you know, once you're off on the wrong track, you know, things could just get worse and worse. ⁓ and so some you know, like unholy combination of adaptation mode, ⁓ local geometry, initialization points, and posterior that would be my hunch. I I think probably like it I don't know if it's something that's systematic or just it would require like a inspection on a model to model basis, but yeah, I'm not really sure. What do you think, Adrian?
speaker-1: Yeah. So about the it failed spectacularly, it's been a while that I've seen one where I would kind of use where where I would say it like that. ⁓ so we definitely had some in the past where there were just some overflows, or you could could somehow smuggle a nun through the tests some kind of through the the cracks somewhere and you ended up with a nun and then everything failed. but things where it works less or kind of just maybe the Maybe kind of you change from diagonal to low rank, and suddenly your tuning isn't long enough anymore. And sud something that didn't actually converge. So you'll get horrible results or something like that. That definitely does happen. And ⁓ I think that that'll just s to be entirely honest, I mean we can fix individual things and hopefully get more of them right, but there'll always be something something that's
speaker-0: Right, yeah, yeah. No, for sure. Yeah. Yeah, as I just was saying, it's a it's a matter of like of averages and I mean not not even only averages, but mostly distribution of cases, right? If if you are able to cover the bulk of the distribution of cases, that's the goal of NetPy, right? It's not it's not a customized ⁓ package for just one kind of model. So y y you cannot really zero in on on these kind of issues. ⁓ something actually Yeah
speaker-1: One thing I've noticed quite a well quite a lot as well is kind of if people compare NutPi and Stan or NutPy and other other samplers, that they it it's relatively common that people kind of compare that, try to compare cases where both samplers haven't really converged well. So where the effective sample size is actually still four in some cases, for for one parameter or something like that.
speaker-0: Yeah.
speaker-1: And I think that's an important thing, kind of how how do you compare that? Because I mean it can still be meaningful, right? One sampler might get much, much closer to the actual posterior than the other one. But if you don't know what the actual posterior is, it's really hard to tell which one it is. So Yeah. ⁓ So if you compare samplers, be careful, really, really careful if they if none of them have converged yet. converged in the sense of decent is have effective sample size for all parameters, not just all but one parameter or something, but all parameters. No divergences and so on. If you have that, then you can compare the samplers. If you don't have that, it's still interesting which sample does better, but it gets much, much harder to figure that out. And to the point of right now I don't know how to do that. So F in the benchmarks, for instance, we just excluded everything where none of the samples converged. We just threw those cases out because they are important, but I don't know what to do with them. so I think that's also an an an interesting thing to keep in mind.
speaker-0: Yeah, yeah, definitely. Okay, so it's like when you're comparing samplers, you you have to be even more careful of having all the green lines, right? Because when you're working on a model for for a project, you don't have to have all the green lines as long as the parameters that are the most important for your study are are having the green lights and if you have some orange lights for some Nuisance parameters that's that's fine. But if you're comparing the samplers, you really, really want to be in the best of cases. And Elliott, actually I'm I'm curious if you if you know about that and Adrian also for you. What I'm the intuition I got about the the diagonal plus low rank adaptation is you do need a bit more time in tuning, right? ⁓ for it to be efficient. ⁓ or is that is that the wrong intuition? I think that's right. 'Cause you need a an eigen decomposition, ⁓ which is O of N cubed, if I'm not that's right, I think. ⁓ Yeah, yeah, so there's a little bit of ⁓ additional like work, I think. But I I mean I think it pays off really well as evidenced in the in the paper. Yeah, for sure. Yeah. Yeah, yeah. The idea here being like kind of amortizing the sampling time with ⁓ a bit longer tuning time, ⁓ which you know that the cases definitely pays off. It's like yeah, it's like eating healthier than you're real quick long
speaker-1: It's not so the if if we're talking about kind of the performance of the low rank one that's not n cubed. So that's a little bit better because of the low rank structure that we can make use of. Yes. If you do an eigenvalue decomposition of the full of kind of a full mass matrix that's n cubed or ⁓ the dimensionality of the problem cubed, ⁓ the low rank one is better than that. But ⁓ it's still it still costs something, so.
speaker-0: Mm. Yeah yeah for sure for sure. Okay, yeah. Yeah but thanks. Good ⁓ good precision, thanks, ⁓ Adrian. And so I'm like to zoom out a little starting to to play us out here because ⁓ well ⁓ starting to get late for you, Adrian. ⁓ so I wanna be mindful of your time, but where where would you say your paper and Which is the the the research we just talked about, where does it fit in the bigger picture of Hamiltonian Monte Carlo research right now? And other other adaptations or preconditioning ideas that you're watching closely ⁓ and and you think have a lot of potential?
speaker-1: So in general, I think there are lots of ideas floating around that are interesting. I'm not sure if we have time to list them all, that because there are many, many cu really cool things around, I would say. But on the preconditioning one, I would say there's not that much from I I I don't want to c maybe I'm missing something important. but ⁓ so there there was a paper that's definitely important, I should add that, that looked like s at something that's a bit similar that to the NutPie mask matrix, but just using the scores instead of scores and and draws. But in general I would say it feels to me like we we kind of have kind of the state of the art there right now. I would say. Maybe somebody else disagrees and I don't want to kind of mess that up in any way but On other parts of sampling, I think there there are lots and lots of really interesting things happening. There's the sparse nuts, for instance, that was going on the on the dis in Standos Discord. There was a bit of a discussion about that a while ago, for instance. That's more about the idea of using more information about the model. So right now or most samplers that are used are really black, kind of Stan and and Py and C at least, are really black box samplers. So they Take the log p function and nothing like log p function and its gradient and nothing else. But there's lots more information in the model in a sense. And it would be really, really useful to use that for preconditioning, for sampling, for all manner of ways. so I think that's a really big topic ⁓ for sampling in general. Then kind of on the HMC side, there's for instance MC LMC. kind of the microcanonical sampler, I think that's really cool. I'm not entirely sure how much better it actually is. It defin f ⁓ I can definitely come up with example where it's better. ⁓ That's really interesting line of work, I would say. then there's walnuts, for instance, kind of the idea of choosing step sizes dynamically, because kind of in HMC as we implemented right away right now, we're always using constant step sizes. And in in some ways that's just dumb. We we we just shouldn't We d we just should say, okay, here we need small step sizes and here we need need large step sizes, which every ODE solver will do. But that's hard to do while still keeping the posterior exact. And walnuts is a way of doing that. I think that's really cool. also I recently came across a boomerang sampler, which is also kind of com the zigzag sampler, kind of that that family of things, if you've heard that. ⁓ that also has an HMC like variant, for instance. And I've been thinking that could also be a dynamic step size HMC in a sense. So that that's cool as well. Then then there's the whole line of work of using GPUs better. So kind of having many, many chains instead of only only few. Yeah. So ⁓ I I'm forgetting things, but the the general gist I would say is of Things I've been thinking about. Of other people's work I've been thinking about. But lots of cool
speaker-0: Yes. Yeah. That that that's great. Thanks for this ⁓ for this list, ⁓ Adrian. ⁓ I definitely need to put a link to Walnuts in the in the show, so I'll do that later for sure. and by the way, for those of you looking on YouTube, ⁓ it's amazing now because of sunset. ⁓ Adrian looks like a black shadow. ⁓ It's amazing because you look like you're, you know, one of these anonymous ⁓ testimonies in ⁓ in ⁓ documentaries. So I I'm gonna make sure to anonymize your voice for that part of the of the episode where like you're giving us the dark secrets of the of the HMC world.
speaker-1: Yeah. all clouds all day except for right now, so that's ⁓ damn.
speaker-0: Yeah, Elliot, ⁓ anything you wanna add on that, you know, like ideas that ⁓ that you're paying attention attention to in the adaptation and preconditioning world? ⁓ yeah, I mean I I not so much ⁓ adaptation and and preconditioning. I know that ⁓ diffusion models are like all the all the rave right now. ⁓ and I think there's a lot of people working on how to use those to precondition densities. I I personally don't know much about about those. ⁓ nor I I'm also not sure if there are actual like software project you know, open source projects for wide use, you know, ⁓ with those implemented. ⁓ yeah, I think I guess broadly what I'm most interested in moving forward is different ways of parallelizing MCMC, I think, and making it ⁓ you know scalable to ultra high dimensions. ⁓ you know, because MCMC is inherently like a very sequential algorithm w while meanwhile a lot of the you know computational advances that we're getting in hardware are for parallel ⁓ computation. So I think that's kind of interesting. There's there's some interesting ideas on how to bridge that gap. But I I would agree that for for preconditioning, yeah, it seems seems like not pious state of the art. Yeah. Mm-hmm. Yeah, I was gonna I was gonna ask you actually ⁓ you know what you're gonna what are you gonna focus on in the in the coming month, ⁓ things you wanna learn in particular or or explore? Yeah, I I'm particularly interested in ⁓ a new way of Parallelizing MCMZ over trajectory length. So there's ⁓ so there's a method of ⁓ you basically ch can solve ⁓ an optimization problem for to solve a full MCM2 trajectory instead of computing ⁓ in sequence, you can make an initial guess for a whole trajectory and then iterate on it, and it provably will converge to the true trajectory, which ⁓ I think could be really cool for problems where the chain actually is ⁓ somewhat predictable to some extent, but the log p evaluations are ⁓ really expensive, for instance. I think it doesn't really work it it's I mean part of what I'm interested in is how that can be done for adaptive ⁓ samplers where it becomes a little more complicated. So that's one thing I'll be focusing on a little bit. Yeah. ⁓ Yeah, sounds like fun. And and Adrian, what about you? Because you always do ⁓ always do some fun stuff, investigating some new ways of doing things. ⁓ so yeah, like I'm I'm curious what's on your mind for ⁓ the coming month.
speaker-1: Yeah, as always, I've got more projects running than I can actually finish, or I don't know. So that's that's always a little bit of a problem, kind of focusing on on particular things that I can actually get done. But ⁓ as I mentioned, MC LMC and kind of the non adjusting methods, that's something I've been looking into. Maybe it's it's entirely possible I'm just doing something wrong, of course. So I don't want to bash on the methods at all in any way. But so far I didn't have too much luck, but that could be me. ⁓ so it's a question of how much I I want to try that more. ⁓ definitely I'm looking into kind of what I mentioned a little bit in the beginning, is of kind of using higher order autodiff ⁓ as well. So not just the Fisher divergence. But what happens if we also use second order derivatives? I think there's really cool theory that could ⁓ could tell us how to do that. so that could be really, really useful. Or interesting. No clue if it's actually will turn up to turn out to be useful, but fun for sure. and then probably looking more into walnuts and this boomerang sampler idea ⁓ than that than I'm Kind of two completely different things, but both I think can be thought of as the the the walnuts foot for sure, but the boomerang as well, kind of an adaptive step size thing, ⁓ which I think could could really help a lot because I think cases where that would be really useful are quite common. and kind of this idea of using more model information, that's also something I really, really like. I've been looking into that a little bit with with Ricardo, where we now have a little implementation that can choose between centered and non centered parametrizations automatically, for instance. Yeah. Kind of moves that into the mass matrix adaptation, which I really like. ⁓ but they have many more ideas in in then. Yeah.
speaker-0: Mm-hmm. Yeah, I think that last one actually is a is a very fun one. I saw it this morning ⁓ in the Discord ⁓ of PyMC. So maybe can you can you expand a bit on that? I mean it's something we've already mentioned the show with you and and and other ⁓ guests, but that's great to actually see an implementation of it that seems to be working. ⁓ so yeah, can you can you tell us a bit more about that and and if people can already try it?
speaker-1: Yeah. Yeah. So centered non center parameterizations are one of those things that if you've seen them once, kind of it's relatively easy to understand how to how to do that. So you ni take a normal distribution, you either say it's sigma equals some val it's kind of the some standard deviation that you want, or you set Zigma to one and then you scale the normal variable up with your standard deviation. And it's the exact same thing, it doesn't make a difference kind of meaning wise. It's boring. But it can really, really influence sampler performance. So if you do it wrong, and wrong means depends on your data set, right? It's not like wrong you need to always use non centered or something, but sometimes you need to use centered, sometimes you need to use non non centered. Basically if you have lots of information, you want to use centered, so lots of data for that particular parameter, and otherwise you wouldn't want to use non centered. So that's a mess. That's no fun at all. But turns out kind of this general framework that ⁓ that NutPy uses to learn transformations, we can use that to choose that automatically. And not just the extremes of centered or non-centered, but there's a parametric family in between, you can say halfway non centered and halfway centered, or yeah three quarters, whatever. And ⁓ that's one parameter that you can just think of as a parameter for your parametrization. ⁓ a transformation parameter. and you can basically treat that as part of a normalizing flow if if you want. A really, really cheap one. Really one that's easy to evaluate. ⁓ so much nicer in many ways than kind of the s the big neural networks or something. ⁓ and you can just learn it during sampling. So the implementation that we have, it's still in a PR, it's a draft, it's not not done yet in in It it works, but it's not ⁓ not deployed in any in any meaningful way. It just looks at a Py ⁓ C model, looks at normal distributions that have standard deviations that are random parameters, and adds kind of parametrization parameters in a sense that say do we want to have this as a centered or non centered one? And then we add that in the mass matrix adaptation and just learn, well, where where where should we place the dial and automatically choose that.
speaker-0: Yeah. Yeah, this is this is awesome because basically the idea would be well the user writes whatever model they want. Usually they will write the central parameterization because it's much easier to code and write and understand for humans. But then once you kick it off once you kick off sampling, you Pinecy would give that to to NutPy and then NutPy under the hood during the mass matrix adaptation would be doing the intro parameterization between centered and uncentered, which would be like a small normalizing flow adaptation, so adding a bit of a cost in tuning time of course, but then ⁓ really rewarding you for for sampling time. And also just making the model much easier to write and and sample from. Yeah.
speaker-1: Also kind of if you do it manually often you need to resample, right? So I'm not sure kind of saying it's taking sampling time is actually true because kind of it's just sampling time you now don't need to spend yes changing your model and
speaker-0: Completely. And you're trying and you're also getting a better scenario where most of the cases, I'm guessing, are gonna fall in between the pure centered or pure non centered. Yeah. Just pro
speaker-1: Don't know yet, but that's an interesting question. I'm actually curious what what how that turns out, yeah.
speaker-0: Yeah, exactly. I mean, I guess from a prior perspective, I think it's easier to bet on the on the fact that the distribution is probably more like inverse U shape instead of U shaped. ⁓ but yeah, that'd be my bet, but I'm ⁓ I'll I'll be happy to hear if I'm wrong. And most of the cases are like pure centered or pure non-centered. ⁓ Also something that's useful, I don't know if you have that already in the implementation, but it's the idea that You don't have also to have fully centered for the whole model or fully non centered for the whole model where it would be like some clusters would be maybe fully centered, some others would be fully non centered, some others would be in the middle. Would that be working like that already?
speaker-1: Yeah, it it's doing that. It's it's doing that already. So it's kind of on a per per variable basis. So if you have one big normal distribution with some crazy big shape, then it can can learn that okay, some of those individual normal distributions kind of inside the big normal should be percented and some should be uncentered, non centered.
speaker-0: Yeah. Yeah, yeah, okay. Yeah. Yeah, which is awesome. It's basically like the case you want where you're able to infer the whole continuous space of that spectrum instead of artificially reducing it to a binary choice, which is what we're doing right now, but just because of the limit technology limitation or maybe just like time limitation. Whereas if you're
speaker-1: I I've actually used some some versions in between. It's not too hard to do kind of manually either. It's just a pain. So I don't think nobody does it because of that.
speaker-0: Yeah, yeah. Yeah, because this is like so many cases to try. So I think it's it's it's a perfect use case for automiz automatization for sure. ⁓ awesome. Well that's that's super cool. ⁓ Elliot, anything you wanna add on that before I ask you the last two questions? Let's let's go ahead. Okay, awesome. Well ⁓ Adrian, you've had already these last two questions. So if you want like you can you can answer them again actually. ⁓
speaker-1: ⁓
speaker-0: But ⁓ Elliot, it's your ⁓ first time, so you know, ⁓ first one is if you had unlimited time and resources, which problem would you try to solve? Like any
speaker-1: And careful it's trick question because it doesn't mean just invasion stats he got me with that last time. ⁓
speaker-0: Well, I would say maybe maybe something ⁓ which problem? I maybe like the clean energy problem? Like how do we, you know, convert w to one hundred percent clean energy. That'd be that'd be cool. I'd I'd probably work on that. Okay. Okay. So I guess the closer closest to that would be fusion, something like nuclear fusion right now or something like that. Hm. Okay. Yeah, well That sounds ⁓ that sounds very interesting and important, so I I love that. ⁓ Adrian, you wanna you wanna change your answer from last time? Maybe you solved already the your first problem, so you you want another one.
speaker-1: So so so the sampler is now actually a solved problem. Okay, that's that's nice to hear. Yeah, exactly. Yeah, I guess somehow fixing the democracy issues that we are facing, but I'm not sure how infinite resources would help with that. Seems more Yeah. That some people have infinite resources seems to be a pro more part of the problem, I don't know.
speaker-0: It's gonna take longer than a few episodes, so I guess when you come back you'll you won't ha you won't have to answer this question again.
speaker-1: ⁓
speaker-0: And second question, Elliot, if you could have dinner with any great scientific mind, dead, alive, or fictional, who'd he be? Maybe does can can I say like Kurt Girdle? I mean I he's not really a scient scientist, I guess, mathematician. ⁓ I I think that that would maybe that would be my ⁓ especially since a lot of the work he did was when he was like twenty two years old or something, so I think. Very ⁓ I I would say him. Yeah. I mean ⁓ great choice and you're the first one to to answer for that, so ⁓ well done. ⁓ Adrian, you wanna ⁓ maybe ⁓ have another dinner ⁓ with somebody else? ⁓
speaker-1: I'm actually not sure who I chose last time.
speaker-0: Yeah, me neither. But we can go back to your ⁓ transcript.
speaker-1: I guess it w probably was Bertrand Russell, so I'm not gonna choose him this time.
speaker-0: Possible.
speaker-1: Maybe let let let's pick Euclid for instance. That that that sounds like fun. If i if we assume that language barriers are not not not not part of the of the thing there. But because otherwise they would be a very very boring dinner if
speaker-0: Mm-hmm. Mm-hmm. Okay, so I will check I will check who you chose, but actually, you know what? I think you picked Euclid now that you you Yeah. I remember you saying something about language barrier or something like that. Yeah. We'll see. But I mean, if you did pick Euclid and then did that again independently, that it means like it's really a re ⁓ revealed preference. So you know. ⁓ we definitely need that dinner to happen. Hopefully, hopefully next time you will have had it.
speaker-1: Yeah. Time machine is almost done, but still kind of the last the the the last ten percent. It's always it's it's always the the l last little bit. The seat is there, the
speaker-0: Yeah, yeah, you know. It's the time machine and ⁓ Yeah, yeah, yeah. Yeah, time re time travel is really an endeavor where ⁓ the Party effect really doesn't kick in, you know. It's like eighty percent of the result doesn't it doesn't get you there. and actually nobody answered time travel for ⁓ the first question ⁓ in the history of the show. So you know. ⁓ I I am surprised actually now that we
speaker-1: That's missing some some some tiny piece.
speaker-0: Talk about it. Maybe it's not that important. That may be why.
speaker-1: ⁓
speaker-0: Awesome. Well, guys, thank you so much for taking the time. I was ⁓ pleasure. Adrian, always a pleasure to have you on the show. ⁓ come back anytime. Elliott, really great to have you for the first time on the show. Hope you enjoyed it. And ⁓ also you're welcome anytime. ⁓ thank you to ⁓ Buck Carpenter also for ⁓ working ⁓ with you guys on on this paper and all these topics. ⁓ I know he's he's doing amazing job. ⁓ at the the Flatiron Institute and and the whole team. So as usual there will be a lot of links in the show notes. feel free guys to add any any links to papers or tutorials or things like that that you think are are worth sharing. And thank you again for taking the time and being on this show. Thanks so much. It's been a blast. This has been another episode of Learning Bayesian Statistics. Be sure to rate, review, and follow the show on your favorite podcatcher and visit learnbasetats.com for more resources about today's topics, as well as access to more episodes to help you reach true Bayesian state of mind. That's LearnBaseTats.com. Our theme music is GoodBayan by Beba Brinkman. Fit MC Las and Megaran. Check out his awesome work at bebabrinkman.com. I'm your host. Alexandora. You can follow me on Twitter at Alex underscore Andora like the country. You can support the show and unlock exclusive benefits by visiting patreon.com slash learnbase tests. Thank you so much for listening and for your support. You're truly a good baby and change your predictions after taking information. And if you're thinking I'll be less than amazing, let's adjust those expectations. We show you how to be a good daisy. Change calculations after taking fresh data. Those predictions that your brain has made. Let's get them on a solid foundation.