Full Transcript for #164 Bayesian Workflow, with Andrew Gelman, Aki Vehtari & Richard McElreath

that's the Instagram account. You can also look for Pierre Simon Laplace on there. So yeah, feel free to hang out over there too. I would be happy to talk with you there. Second announcement: I will run a Bayesian workshop that will also double as a Learn-Base Stats episode at the Carnegie Mellon Sports Analytics Conference in Pittsburgh, Pennsylvania on October 23rd. And also the Conference continues on the Saturday, October 24th. I will of course be there. So if you wanna come meet me and also attend my workshop, that would be awesome to see you there. I will probably teach you something about bass, I'm C, and take a soccer model, maybe that you've heard about from me, to teach you all of that. So, yeah, the idea will be to teach you how to causally infer the effect of a player and the effect of a Team and how to disentangle them with one model. I will give you more detail later, but you can already sign up on the page of the conference that you will find in the show notes. So hopefully, see you there on Friday, October 23rd. And also a big thank you to Ron Yorko for inviting me and the podcast over there. You can find out more about Ron and his background and his research actually in sports analytics. in 140 of the podcast. It was a very fun one. I definitely recommend it. And now, without further ado, let's hear from Andrew Gellman, Aki Vetari and Richard McElreath for a very special of this podcast. Andrew Gelman, Aki Vetari, and Richard McLeath, welcome to Learning Basion Statistics. speaker-2: thanks. Glad to be here. speaker-1: Yeah. Well Aki, Andrew, you know your you know the place now. maybe a different software, but you know the show. Richard, first time here, so it's it's a real pleasure to to have you on. and we'll start a bit with you, you know, because like you're a The new kid on the block for this episode, even though I'm sure everybody knows you, who is listening right now. but I'm actually curious because I haven't heard too much about your origin story. so I think most people know what you're doing nowadays. and mm what I think is that maybe not a lot of people know how you ended up doing that. So can you can you give us a bit of a bit of a an origin story of Richard? speaker-2: Okay, I'll I'll give you a short origin story. I say first, yeah, thanks for having us and me on the show, longtime listener, first time visitor. yeah, the what's my origin story? I guess now most people think of me as a Bayesian statistician. That it sort of dominated my career now. But I'm technically an anthropologist, an evolutionary anthropologist, which is what my current job is as a director at the Institute in Leipzig. And I I stumbled into that too. So my career is just a series of stumbling into things that I need to do. And but when I started college, I didn't start off in anthropology. I was doing actually I did a year of music theory and composition, my first year in college. thought maybe I want to be a music teacher. And then I decided

I was just too average at it. Right. So I mean, I enjoyed it lot, but y it's extremely competitive and just too average. At the same time I was taking ancient Greek because I had had Latin in high school and I really enjoyed translation and and I enjoyed taking ancient Greek, but I decided there was no future in being a classicist either. sorry to classicists listening. and and I randomly took an anthropology course just as a distribution requirement and thought it was it's and it's extremely interdisciplinary field, so it was all the things I liked in one place. I say also in high school I was on the math team and I was pretty good at it. but I didn't I didn't particularly enjoy math, so I didn't go into math in college. But that has turned out to be useful because I as I went on in anthropology it turns out I could go into the say statistics journals and understand what was going on and translate things and and read papers and and so on. And my my introduction to to stats really, where I got more serious about it, was in graduate school at UCLA and I took the the School of Public Health there teaches a quite intensive year-long course for their their mat I think master's in public health students is what it was designed for. But it's quite good and has a bunch of real issues about design and survey and dropout. Heckman corrections and all those practical things and a bit of causal inference stuff, because of course there are some famous causal inference folks on the UCLA campus. and it was a great, great course. I learned a ton from it. And my PhD supervisor, UCLA, is also an anthropologist, and he's a theoretician, and that's sort of what I went to graduate school to do is to do evolutionary modeling. And but but he's Rob Boyd's name, he's a well known evolutionary anthropologist. He doesn't do statistics really. so his advice to me was when I asked him how to I had collected some data and I wanted to fit it to a theoretical model and his his advice was, no, go go get a book on maximum likelihood and I think that was good advice in the sense that like actually they get you pretty far. I mean this is maybe sacrilege for a Bayesian podcast, but it kinda gets you pretty far and it gets you on the Bayesian course, I think. And so yeah, I I went from there and I I ended up reading a bunch of books and including Andrew's books, of course, at that time. And and Janes. I found Jane's, which of course Jane's is a mixed bag, but it's very entertaining. It's sort of the most interesting and and I like eclectic, heterodox things. So that's how I sort of got into it. And then I got a Well, you know, my career, I moved around. I had a few a pre few professional jobs. I eventually got a job at UC Davis, University of California, Davis in the anthropology department. And I was a, you know, first year untenured faculty member and the chair of the department heard that I was good at math. at that point I had written my first book. I I hadn't published it yet, but I had a manuscript. My first book is an evolutionary game theory book. And he's like, I hear you're good at math. We need someone to teach statistics. So you're gonna teach statistics. And I was Yes, sir. 'Cause you know, I want a tenure and but I really had never taught stats before and I wasn't sure how to do it. So, you know, I I started trying to develop a curriculum. I I got a lot out of the old regression, multi level regression, Gelman and Hill sort of approach. That's sort of where I started from and then adapted. Yeah, and it's a classic. Right. And no, it's got this pragmatic style. I mean to all joking aside, it it stood out, especially at that time in Pagan's very pragmatic, applied style, very interested in the data and the scientific background and all the examples. And and that really stood out, I think, as as Stats book goes that they're gonna teach multi level modeling. And there's many, many more books now that were of course inspired by that book, I think, including mine. And then I started adapting the curriculum. to my students. And one of the things that that was important to me was that when they work with the software, that they are really writing every line of the assumption in the Bayesian model. So I wrote my R package for that reason to be maximally annoying. So that they would have to write every line, which I'm not against convenient software. The formula interface like in BRMS is great. But for teaching I I wanted a an interface that's much more explicit. But isn't all the way into like raw stand code. Now, of course, stand didn't exist at the time, right? Or it was a twinkle in Andrew's eye or something. And and I didn't want to have him write in bugs exactly either, because I wanted to do simple, start with simple like maximum posteriority stuff that would run really fast. So I wrote my R package with that purpose in mind, just as a teaching tool, and it kind of cascaded from there. And then I should say that you know, like everybody's career, I think, is just lots of lucky coincidences. The stats department at UCLA has some people in it who are editors of that series, the the red cover series, and they kind of heard gossip on campus about my course and it gotten popular and and the book kind of fell out of that and got into that series that way. And it's a great series because you get a lot of you know, it's in the same series as BDA, Bayesian data analysis. So you get a lot of free press by being listed with it. And that helps a lot as well. anyway, I think that's kind of the story and and the stats thing is I mean, the I think it it's really taken over my career almost I mean, I'm probably twenty percent an anthropologist and eighty percent a statistician now, which is very rewarding because there's just a very large interdisciplinary audience to to benefit from my efforts. anthropology is a very sm small field. I love it to death and I'm not gonna give up on it, but you know, there's it's very, very small.

So the the impact, the effective impact from work on stats is is been very satisfying to see how much people get out of recorded lectures and things like that right. Anyway, I'm being a bit long winded, apologies. But yeah, I just stumbled into this and we'll see what I stumble into next. Maybe I'll start a podcast and speaker-1: Yeah. May yeah, yeah. If you if you wanna take over, you know, I could I could use some holidays, yeah. So speaker-2: We can try that maybe sometime. speaker-1: yeah, I mean your your career is really super interesting and I remember talking about it with you last year at StanCon. Like this year we we need like I need to get it again on the show. Maybe at StanCon, we'll see, you know, like not announcing anything. But and yeah, like talk a bit more about, you know, anthropology and and how you do that concretely. I think it's super interesting. It's gonna be super concrete for listeners and also because anthropology I read is is extremely interesting and so fascinating. so definitely something we need to do. but today we're gonna focus a bit more about I think you three have a book coming up. that's that's what I heard on you know on campus also like just whispers. and so I wanna talk about that but first yeah how how did the the idea the idea of the book Come up. who like did one of you had mainly the idea of contacting the other one or the other ones or was it something like spontaneous? speaker-3: What happened was that we we had the you know, Bob Carpenter wrote the Stan manual, the users in the Stan User's Guide. And but th th that's kind of written it's great, but it's written a kind of computer computer manual style. Like it it has little snippets of code, but it doesn't really have examples and stuff run from beginning to end. And so we were talking about having a a book, a standbook. and then I made some draft, this was almost ten years ago. then it somebody put it on GitHub and then everyone was supposed to help, but then like nobody does anything or people do arbitrary things. It it was like nothing was happening. So we decided just to have a Bayesian workflow book and not like not tie it to the Stan project just because Stan is great, but i any project gets paperworky and like then you have to go through certain rules and then you can't actually get anything done. So like I don't want to have to have to do a pull request to like add a sentence to a book. So yeah, and then So we had this idea of writing the book and then a few years ago we wrote an article. The purpose of the article was really just to get us to start the book. So we wrote the article and then stuck it in an archive and then the original idea was roughly each chapter each section of the article would be a chapter of the book and then it got re we reorganized it. And we have eleven authors. and the other eight authors have done a lot too. So it's like We the three of us were kind of kept the project moving and basically took responsibility for

all of it. So all three of us read every word. And speaker-2: But speaker-3: yeah, and then speaker-2: What? Rewrote every word. speaker-3: Well there we wrote every word. There are parts of the no, some of the case studies were were written by other people and original versions and there were taken there were bits taken from the article that had been written by co authors there. So we didn't literally write every word, but we take responsibility for it. And then Aki put together the with with Jonah Gabri, put together the website which has all the code and I mean, there was an earlier version of the book where the code didn't all run, yeah, kinda like my other books. but this time we made sure it all it all ran and so that's pretty exciting too. speaker-1: Yeah, yeah, definitely. And so these links are in the are gonna be in the show notes for this episode. both the the the the website from Aki and Jonah and and the original paper. So folks feel free to to look into that and also of course the link to to buy the book. if you wanna if you wanna read that. I'm I'm pretty sure if you're listening to that you wanna read the book. and actually can you Give us guys the you know, the elevator pitch of the book. What what are you trying to do with that book and why did you think it was it needed to be written? And does does it have a particular audience, like the the who of the book if you want? Who who wants to take that one? speaker-3: Aki, you wanna take that one? speaker-4: Yeah, I think that we wanted to put there everything we know that's not already in PDA tree and statistical rethinking. speaker-1: Uh-huh. speaker-4: Plus of course then these case studies that instead of just saying like giving recommendations what to do, actually saw that in practice that it it's you can follow For different data sets problems, the steps from the beginning to the end. Of course, different case studies are also focusing on different aspects of not all case studies go through all the different workflow steps we describe in parts one and two, but in a way that each case study shows actual steps what happens what we can see from different steps and diagnostics and so on. speaker-3: Mm-hmm. And I see a like to me there's a big picture that which which is mentioned in the book of of a spectrum. And on one extreme there's examples and on the other extreme there's theory, and this is somewhere in the middle. So the idea is that like you have theory, like mathematical theory, like in Bayesian data analysis, here's Bayesian inference, and you can figure out the the answer. And then there are methods, like specific models to use and and and so forth. Then on the other side you have examples and case studies. And to us, workflow is something that I think it's more codified than a case study, but less formal than a method. Like maybe everything would like to be ultimately a theory. And before it gets to be a theory, it would like to be a method. But these are things that it

wasn't quite like it's not quite an algorithm or a method, but it's things that you would want to do. So maybe the most important idea we emphasize in the book is simulation based experimentation, fake data simulation. So we I mean We use simulation to do posterior predictive checking, and that's an old idea. That's in Bayesian data analysis and and prior predictive checking, but just kind of more generally the idea that you understand you can understand a process by doing some simulation. If you fit a model to data, the first thing you can do is fit data from the model and check that your procedure or see if your procedure can recover things. You learn so much from that. And that's a that's something that's It's not really in our other books in that way. Like we have examples where we do that, but the idea that this is sort of a a very this is a technique you should do be doing all the time. Then there are more specific things. iterative like iterative algorithms, the starting point is very important. That tends to be hidden either because people are kind of embarrassed about it or they think it's kind of cheating. So we have a little section where we just talk about the importance of starting points. We have a little example. That's kind of a principle. shoot, there's another one of these things I was going to mention. now I'm forgetting. Anyway, di different things come up, like certain we have a long section about priors. BDA, we have what we call the Bayesian cringe. That it's like sure historical thing that we are embarrassed to use prior distributions. my book with Jennifer also, it's the same thing. We have a lot of reassuring the reader that no, you don't really need a prior. With a hierarchical model, you could just estimate the hyperparameters from the data. Over the years we started to realize that well actually even with a hierarchical model you can get a lot from prior information. And that's really part of the workflow also. And so we put a lot of stuff in there that's not in Bayesian data analysis about like ideas like how to think about priors for effect sizes and a lot of stuff we've been thinking about. the term workflow itself I really like. I I don't know where exactly we picked it up. I think it's in the air in engineering. and it just it's it seemed right. I think there's a little confusion because it's not a It's not a checklist, like a workflow, like the workflow if you need to repair your air conditioning unit, that kind of thing. but it's a set of tools. speaker-1: Yeah. Richard? speaker-2: Yeah. I I think my perspective on the book is that it it it meets directly the kinds of questions I get from colleagues and students much better than the other well say my other book, which I also like to think it meets that, but but this book is much more for people who already know the basics. Like is this isn't a book to teach you Bayesian inference, but it's to help you become more professional about it and have the ability to make theoretically informed justifications for the particular branching points that are present in realistically complicated analyses. lots of books, including mine, have really simple curated examples for teaching. They're like the data sets come clean in the example, and then you go right to one part of the workflow that's relevant to that example. And that serves its purpose in an introductory course or someone's first course in Bayesian modeling, but for actual practice, for people who end up doing this for a living, like yourself, you know, the real real workflow is much more complicated. And there's a bunch of decisions. And those decisions, when they're not presented in a in a book, it it seems like they're just arbitrary. But of course there are principled, theoretically underpinned ways to make those choices, and we wanted to present some of that. And I think the the the book's very unique in that regard. as being much more of an engineering approach, if I can use that word, maybe Aki appreciates that. But much more of an engineering approach. and respecting the idea that there are lots of little decisions along the way and you should feel free to explore them with as Andrew said, simulation based inference, but also that there are principal diagnostics and ways to inspect, say, The strength of priors and and those things as well. And so this is a a book that lays that stuff out in a clear way. It also has is already ended up being useful to me professionally because lots of the times in collaborations or just colleagues will write me and say, I have this manuscript and I need to cite weekly informative priors, like the justification for that. And so this book is is basically ideal. It's the new state of the art for having a mature reason section on say weekly informed priors and what they really mean and why we prefer them as we do. And and there's other things too, various cases. I f I forget there was an example that that came up earlier this year where a student was Done some trick that I had taught her and then needed to cite it and I was like, no. And it turns out it's in the book. And I think Andrew wrote it in there actually and not me. I forget what it was, but it was some nice trick for solving a problem. And so it's full of stuff like that too. It's got it's got use. You can the first half of it goes through steps of the workflow and lays out theoretically informed diagnostics and principles of choice and uses thin examples to motivate those. And then the second half is the case studies where they get applied in the context of a full example, often a quite complex example that weaves together theoretically derived generative modeling with estimation. So I think it's it's also nice in the sense that the examples are much closer to the sort of realistic complexity, at least some of them. Some of them are simple, but a lot of them are quite complicated, especially in going beyond off-the-shelf models to something that's really

generatively bespoke to the scientific background, which I I won't speak for my colleagues, but I think they agree is that that's the thing that attracted me to Bayes in the first place was the ability to take in all of that scientific background I have as a professional scientist and express it in in a joint probability distribution. And some of the examples are like the golf putting one is is really nice in that regard. Aki wants to say something. Yeah. speaker-4: Yeah, but you said reminded also that like scientific papers have the final model, everything just works. Like so all so only the final model, so that all the diagnostics were great. And many of our case studies then illustrate problems, something doesn't work, and how we then go forward from that. How we diagnose the problems but also fix the problems and how this kind of iterative process is okay. speaker-1: Yeah. Yeah. I mean I I love that so much in in in your own research, Aki, and also what what you folks are doing at at Alto. And I think that's also why I have so many people on the show who came here to to talk about their research because it's really not only about the final output but trying to help people along the whole workflow and the whole sausage making. Which is actually the hardest part. Like d which is always something I say people who come and ask me about how they can get started into that is like I'm always telling them to try and find something they really care about, a problem they really care about, because they're gonna have all the models they're gonna work on are gonna fail except for the last one. And even the last ones you're the last one you're gonna be like, Yeah, it's it it works, but they already know how to improve it. You know, so it's like always the final output, something you're happy with but speaker-3: One thing we like to say is that even if you knew ahead of time what model you wanted to end up with, you'd still want to be fitting simpler models to understand what you got out of fitting the more complicated model. And you'd also want to fit a more complicated model just to see that you didn't get more out of that. And a key part so Bayesian data analysis, like back when it came out in nineteen ninety five, was kind of revolutionary and I called it Bayesian data analysis rather than Bayesian inference, because we our big thing was that there are three steps. You build the model, you fit the model, and then you check the model. And then you can expand it. When before that, people really didn't talk about building the model and they didn't talk about checking the model. They pretty much only talked about fitting. But the th Bayesian workflow or a statistical workflow goes beyond what we had in Bayesian data analysis because that's all about you fit the you build the model, you f you fit it, and you check it, maybe you expand it. But a lot of workflow is going from model to model. And a lot of the operations that we do in statistical workflow can

be framed as fitting additional models. Even gathering new data could be thought of as fitting a new model. data in the sense that if I have a bit of data and I fit a model, then I get more data and fit a bigger model. You could think of the original version as a very simple model fit to all the data, which just didn't use the new stuff. So thinking about our workflow, I mean it has a lot of implications. So one implication is like how do you like if you have an iterative algorithm, how long do you run it? So some people say, well just to be safe, run everything overnight. Well that doesn't really work with workflow because that will reduce the number of models you can realistically fit. And we learned so much by fitting model after model after model. And then being able to simulate from a generative model means that you can well let's say this. If you think about like I like to say that like all statistics is frequentist, including Bayesian, because we're averaging over the prior, which is a a frequency. Think Ben Goodrich doesn't like when I say this, but I I I think this is basically right. But if you think about in old school textbooks, what is frequentist statistics? You use a method and then there's a mathematical theorem saying that it has certain statistical properties. But then in the last 50 years, there's been a lot of papers where people have they can't do that, so they do simulation studies. So lots of papers by like Stanford types. Which will have tables at the end with a bunch of numbers that look like point nine three, they're like between point nine three and point nine six, which are supposed to demonstrate the coverage of their ninety-five percent intervals under various conditions. But what's really cool is that you don't need the I mean, the theory helps because it gives you some mathematical intuition. But you don't need the theory and you don't actually need that published paper. Like if you have a if you're gonna use a method, a model. Traditionally you would say, well, let's use a method or a model that's been published somewhere, and then let's hopefully somebody wrote a paper demonstrating that it has good frequency properties. But now what do you do? Well, I'm gonna fit a model and it's a new model, never got created before. I can simulate from the model and I can check that at least under ideal conditions, how well can it reconstruct, how well can the it parameters can be be estimated? And so I can build this tool. giving me more confidence in my own work. So we try to demonstrate that to people that what's necessary, what's important is not just to get the good answer, but to get an answer that you know is good or you have some degree of confidence that's good. And the idea that people can we can empower people that as a user, you can check your own model and I think that's not just the fit to the data, but check its statistical properties. That's super important. It's not really something that we had been saying before. We had been living it in some way, but but writing it down makes a difference. speaker-1: Mm-hmm. Yeah. Yeah, completely agree. And and I am very impatient to get my hands on on this book and and see what it looks like and and the content of it, I can tell you. And I'm sure all the listeners here are. And actually thanks to your editor, everybody listening get a a discount code that will be in the in the show notes folks. So definitely use that if you wanna you wanna read the That book, it's gonna get you directly on the site of the editor. And also actually before the show, I ran a small context for people to, you know, like basically give me other hardest Bayesian workflow problems. I guess it's like, you know, it's my French laziness. I didn't wanna talk. think too much for this episode, so I was like, I'm gonna outsource the the thinking. no, so kidding aside, I got a number of great contributions from a lot of people listening to the show and curious about the book. And one of the one of the main ones was about bats. So modeling bats and and and trying to make sure that like That your model works, and we're gonna get the the person who actually won the contest on the show in a few minutes. She's gonna join us. Her name is Dorota, she's from Poland. and so when she joins, I'll I'll have her present her her problem to you folks and and hear what you would what you would tell her. But before that. I wanted to ask you also because you you write a lot, you teach a lot, so I'm actually curious in this book, is there like what's that's gonna be a hard question, but what's the main thing you would say, the main insight that you added to the book that was something that was new for you in your teaching, like a new insight that you got and you thought was very important to put in this book? Maybe retur it if you w if you wanna if you wanna start with this one. speaker-2: Yeah, that's a good question. I think it's really as simple as just being much more explicit about the structure of the workflow. And so I just have an updated set of of slides for my course that I taught in January this year, starting in January this year, and it does have a lot more workflow diagrams in it. And the feedback I got from students is that they really enjoyed that. And it it seems like a simple thing, but it's not. but to actually draw out the workflow like a network and make it distinct from just a pipeline. I think most of your listeners are probably familiar with the idea of a pipeline. there's similarity between a workflow and a pipeline, but a workflow is much more dynamic to start with. And 'cause there's engineering parts involved in it where the pipeline is something you deploy. Right. That's Maybe you have to fit the model every day, like in your line of work, right, Alex? And you just something's in production and and so on. But that isn't what I'm talking about. And so the workflow is different than the pipeline and in that sense, that there's more

dynamics and we explicitly label decision points and the diagnostics that we might recruit at those points. But also that it's not just steps, but it's also justifications. It's reasons for how why the inputs were combined in this way, like why this model, given the combination of the question and generative model, for example. And then all the way towards the end of the workflow where we're doing the summaries, and the summaries again are justified in light of the questions that we want to calculate. We're going to do these set of marginal effects because they're the right answer to the original motivating question and not something else. And I made a series of diagrams for that. And Students really enjoyed that a lot and I think they learned way better from that as before. And and it's interesting that just that change in the visual representation and the rhetoric about it had I think a l a strong effect on learning in that regard. personally, in my own research, writing the book was very stimulating because a lot of the book actually is unsolved problems in workflow. Like you many of the sections, like here's what we know now and here's what we'd advise and this is why, but this is an open problem. We need more work here. There's a bunch of places where we can think about good components of the workflow, but it's just inconvenient to do it right now. And a lot of those have to do with with things that which we might call reverse bays, which is an old idea going back decades, but the idea that you might want you want to vary components of the model. So reverse Bayes and I think this is a term from IJ Good originally is the idea that you you you solve for the prior rather than the posterior. You say you could start with a model and a posterior and a data set and say what prior what range of priors justify this. So we think of this as prior sensitivity analysis. And Aki's prior sense package is is a huge advance in the convenience and transparency of doing things like this. But there's lots of innovation to do along those lines. So what I call the reverse base approach as essential components. speaker-3: If you wanna formalize that, I I mean I hadn't heard about that before now, but if if you want to formalize that maybe the way to think about it, I mean there's two ways of thinking about the prior. So one is the prior is the range of problems for which the method is intended to be applied or to which the method gives reasonable answers. But another way is to say the prior represents the population. So I wonder whether what you're calling reverse Bayes could be formalized as just Bayes where you're trying to do inference for a population distribution. I mean, that's like hierarchical modeling. So you're saying like usual hierarchical modeling, we had the data and from the data and our our data model, we can make inference about the population distribution. Now you're saying something different. You're saying something more like this is our posterior. So what should the prior B, but I guess I guess the statement this is our posterior, probably there's a way of framing that in terms of data, is my guess. Could speaker-2: Yeah absolutely. speaker-3: be sort of as say, because I I think I IJ Good had a kind of annoying writing style. i I think I mean he has a lot of interesting ideas, but I'm saying like to call it like this alternative approach, like maybe there is a way of of framing it as a speaker-2: Well, as an o as it's an open problem to be refined, I would say that I remember his example was something as simple as, and you've done this, I think, Andrew, and and and I hope you'll accept the the the charge. And you take, say, a non-Bayesian analysis of maximum likelihood and you say, what priors make this sensible? I've seen you do this a bunch of times that show that the priors which would make that conclusion sensible are just not scientifically reasonable. Yeah. And that's that's kind of the reverse phase approach, is what range of priors are compatible with this conclusion and are they also compatible with the scientific information we had? And I think making that an efficient and routine part of workflow means that we need to do some more theory and formalization, as you suggest, and figure out what the boundaries are, but also some tools. We need tools for people to pick up and and make it convenient so they don't have to refit models over and over again. some important sampling, right? Aki is always a useful thing to have on hand. but I think there's a bunch of open problems along these lines. in in addition, there's a bunch of stuff in the book that I quite like, which is about even when you know the model you want to use for inference, you need to build it incrementally, not just for the for the understanding regions, as as Andrew said earlier, but also just for software engineering, responsible engineering reasons. You you do feature engineering incrementally. And having good advice about that is I think still there are open problems there. For a complicated hierarchical model, it's it's there's a bunch of paths you can go to build it and and there may be trade offs involved in that in in how it works. And I have this tendency to start with what I call the empty model with just the random effect structure. And and and get that working before I build in any predictive variables at all. But I have no justification for that except that's the way I do it. And I'd I'd like to have some theory or some set of examples that we could develop that better and think about how this works. Andrew has some interesting ideas about model hypergraphs which relate to this, I think, and what we learn from models which overlap in their features and So there's lots of open problems to do that that came up to writing the book and so we want to invite readers to help out with these problems or or nominate new ones for us as well. speaker-3: Well was also fun to write because we could just kind

of pour ourselves into it. so just for example, on page six of the Bayesian workflow book, right at the beginning, section 1.1 is called why base. And then we have a section starting on we have a subsection starting on page five called on the borders of Bayes. And then on page six, we talk about Bayesian interpretation of non-Bayesian methods. And we I mean, we start, we say a sometimes annoying habit of Bayesians is to take non-Bayesian methods and give them Bayesian interpretation. For example, maximum likelihood is just Bayesian inference with a flat prior, fixed effects are just random effects with a group level variance set to infinity, and and a few other examples. And then we talk about how that attitude can be valuable but isn't always quite right. Like it it's very refreshing. I feel like when you write journal articles, you're under you're under such pressure to never admit that you could be wrong because the reviewers will will jump on you. And so I yeah, we talked about a lot of this stuff. The the other thing I wanted to say is if you have a paperback version of the book and a pair of scissors, you can if you cut it at around page two hundred, the first two hundred pages are Bayesian workflow and it's this crisp little book. And then the next three hundred are the case studies, which is like a bunch of case studies now. There's no point in making it two books because the case studies help you understand the workflow, you know, blah, blah, blah. But if you think of it as a 200 page book, then I think we're really covering a lot and with a lot of open questions. I I at one point we're gonna have a s an appendix. We have some appendixes. One of the appendixes is how to If you're not a Bayesian, what you can get out of this book, like which we think there's a lot, like simulation's important, a lot of things. There's another appendix on how to get the most out of Bayesian data analysis, our earlier book, and what what sections you can skip now, because they're out of date. but we were thinking about adding a chapter an appendix, a list of open problems. But then we decided that like The book itself was like a list of open problems, so it didn't seem to make sense. speaker-1: Yeah. Yeah, yeah no for sure. So I need so two things. I need a pair of scissors now. I need to to buy one. and and second, yeah, these appendices yeah, I I really like that idea. especially the if you're not a Bayesian one. although if you wanted to provoke you could just make it one sentence. If you're not a Bayesian, become one. And and and then you're gonna you you're done. but yeah, so we'll get back to that. keep that in mind. I have still so many questions for you about the book. But first we've got Dorota here. So let's let's welcome her in. Hi Dorota. You're joining live. You're joining live from Poland. So first, welcome and congratulations, you are the grand prize winner of the LPS contest. speaker-4: Thanks a lot. speaker-1: Yeah, no, it's very great to have you here. so you'll get you'll get a a signed copy of the book by the three auth authors here that you're seeing. but today you're here because you're working on some really interesting stuff. So yeah, maybe first tell us a bit about yourself, you know, what what you do, how did you end up being interested in in in Bayesian stats, and and then we'll talk about your your specific problem. speaker-4: Okay. so I'm I have medical background and I was working as an assistant professor at medical university for about eight years, but I finished that and I started to mm work as data analyst. But I'm a junior. and well Last year I knew that I will just resign from my previous job and wanted to start a b Bayesian statistic course and I took statistical rethinking course and in November probably I'll take also another course in Uppsala in Sweden. And I would like to thank guests for choosing my topic and I also can't wait to read the book because I'm waiting for the case studies. and I would also like to thank Alexandra for the support because I have no experience with podcasts. So I will continue with the topic. so I'm building a Bayesian model of bat mortality from post construction wind farms. monitoring studies and this project grew out of my short placement at Forest Research Institute's Vertebrate Ecology Lab. And I'm preparing the dead data set f dataset from these studies. It's zero inflated and the challenge is that observed carcass counts reflect not only mortality but also the detection process including carcass persistence searcher efficiency and also search design. So when I observe zero carcass counts, I don't know whether it reflects low mortality, low detectability or both. And many studies are missing parts of the detection process. So I was thinking to model the detection hierarchically. To allow studies with complete detection data, to inform studies with missing detection data. and I am still in an early stage of model development. That's why I'm trying to understand how much information can be shared across studies without letting the model assumptions dominate the inference. So my question is. At what point does hierarchical pulling of detection parameters stop being principled borrowing of information and start becoming an assumption that dominates the inference about mortality? speaker-1: Fantastic. Yeah, thanks thanks Dorotta. I think that was that was very clear. so I'll just restate a bit and then you guys can take it away. But I I really love that question because I think it's super concrete. But it's also very interesting. It's like so basically the question is like how many how many bats wind turbines kill where you only observe carcasses that persist and get found? And fully insearchable ground. So I really love that pyramid pyramid setup basically here with non missing randomness, which I'm sure Richard you you're interested in. And also the core question that you're asking is a very concrete one that many people I'm

sure get into their own workflows, it's at what point does hierarchical pooling stop being principled, borrowing of strength, and become an untestable assumption that quietly dominates your inference? And and how would you even detect that? So on that note, who who wants to take your five at it? speaker-3: Let me throw some general statements out and then I think maybe Richard and Aki will have more sp specifics. So first, I think that assumptions are important and it it's very it's very rare to have a problem where assumptions don't matter at all. like if you're in a situation where the true parameter could really be ten to the fiftieth, like if that could be, then like your method would have to change to allow for that. And so in some way, I think a a failure of some of my earlier work has been this idea that, a hierarchical model is this innocuous thing and it will always work. but I I think now that it is very valuable to have contextual information. that said, even a week prior can be useful. So to say I have a certain parameter that can be somewhere in some range. it's not going to be less than point one, or if it is, it doesn't matter and it's not going to be more than a hundred or ten. that kind of week prior up can be enough. and it is an assumption though. Like I I think that it would be that The the idea of the purely data-based approach that will work for any value of the underlying parameter, there's only a very few problems for which that happens. And I think we have to move ourselves away from that particular expectation. The other thing is with hierarchical models in particular, there's something the epidemiologist Sander Greenland told me many years ago, like 20 years ago, I think, he said, well, he likes hierarchical models, but he'll just actually set the group level variance parameter to a fixed value based on his priors. And I naively said, but what about, you know, hierarchical modeling? We can get inference from the data. And he said, sure, but often you don't have a lot of groups. You don't get a very precise estimate of the group level variance from the data. You kind of think you know more than you do because you have this posterior, but Often you have quite a bit of prior information too. So I think yeah, he would say the best thing to do would be to use an informative prior and do a hierarchical model. But he would he also said that in practice, often you have enough prior information about the group level variance that setting it to a fixed value won't be so bad. related to that is that we often will have a model we fit to data, then we fit the same model to another data set, the same model to another. And this happens in political science all the time. You might want to fit a big time series model to all your data, but realistically you might fit a separate model to each pole or for each speaker-4: Yeah. speaker-3: data set that you have. if your statistical inference for the hyperparameters is not well regularized,

so you don't have a a strong prior on it, then your inference for your hyperparameters will jump around from data set to data set. And this is something I don't know if we really explain it so much in the book, but imagine you have data and you fit Hierarchical model, and you have a fair amount of uncertainty about your group level variance parameter. And you're like, that's okay. I'm Bayesian. I have a bit of uncertainty. I integrate out over it. I get inference for my parameters of interest. That's fine. But the thing that's funny we don't always think about is that uncertainty in one inference kind of maps to variation in one in repeated samples, new inferences from new data. So that means if I have this broad posterior for the group level variance parameter centered at a certain point, you might feel as a Bayesian's okay to average over it. But if you had a new data set from the same process, it might be a it would be moved over a lot. Like if your posterior standard deviation for this group level variance parameter is 10, then that also means that your point estimate, roughly speaking, could be 10 higher or 10 lower in a new example. and so that will create inferences that jump a lot from one data set to another, and which isn't really what you want. And so the in that case, a strong prior would be would help on that. So I guess my my very general messages are first it basically yes. There it the prior is gonna matter. I think you don't think that you're gonna be in a situation where like it's innocuous and the data tell you the answer. The data will tell you the answer, but within a certain constrained space, you know, within a certain playing field. speaker-4: Mm-hmm. speaker-1: Damn. Fantastic. Thanks, thanks Andrew. Maybe Aki, you wanna you wanna chime in and then Richard? speaker-4: Okay. yeah, this kind of data collection problem of that we don't know the effort and yeah, they also not have the the carcass problem too, but like this effort problem is common in ecology, epidemiology, clinical data where you kind of to collect from the registry data and so on. like the One possibility is that there is just no any information that can be used to then estimate the effort or the this like disappearing carcasses. And that's the that's the kind of the where you have least amount of information, but you can still look how sensitive your results are. to different assumptions that what if the effort varies in certain range? So this like the detectability. Then the better possibility is that if for each site is it possible to somehow estimate the this detection detection like effort plus the disappearing carcasses. That's then you then it goes to kind of similar models that Andrew has used is multiregression post-ratification that you can try collected seeing the similarity of different sides, assuming that that similarity then connects to also similar if art and similar Kakas detection rate. speaker-2: Is it my turn? speaker-1: Yes, yes, and then I I think there is also a hidden a hidden subtopic in in Dorta's questions question that I wanna ask Aki afterwards because I think it's his his ballpark, but please Richard go away. I think you're gonna have great feedback on that. speaker-2: all right, I'll try to I'll try to be concise and brief because I know there's no more stuff to discuss here. Let me try a different tech, is is just very practical stuff with with modeling problems of the type you said you suggest, Dorota. I also I collaborate with ecologists a lot, I think you know, on on these sorts of things. Is as a workflow approach, I always begin with a big generative simulation of the modeling problem that includes the measurement process. Which is is as you know in ecology, it's it's the huge part of the modeling. The simplest part of ecological modeling is is the occupancy model and it's already a complicated model with all these problems in it. And so I always begin before I even think about the statistical model, just trying to simulate data that has the structure and then I talk through the with the experts the assumptions of the simulation and we work through the graphs and try to figure it out, just as is the forward workflow, is the momentum. and get that in place and then take parts of of the overall generative model and try to develop statistical models to estimate the quantities of interest and and approach this this partial pooling problem that you've you've brought to us that way. And I think I agree with Aki in the sense that there's there's scant information on some of the important aspects of the measurement problem, which is very common. So don't feel like you're alone in this this is I think this is most of the applied ecologists I know or deal with these have this kind of similar situation. You wanna do some kind of s you're gonna end up with some kind of sensitivity analysis where you make different informative assumptions about in the statistical model And look at how much pooling that does, and then compare that to the ground truth from your simulations. So lots of profiling and calibration using synthetic data or fake data is sometimes called, but synthetic sounds nicer, doesn't it? It sounds more scientific. That's the workflow I use in these problems to help me forward. And then the target ceases to be a single posterior. But a set of posteriors that give a range of estimates under different scenarios. And the core problem you've brought to us, I think, is a very common one is partial pooling is this incredible technology, but sometimes you don't want the outliers to be shrunk in because they're information. especially you're sort of on the bleeding edge of an applied problem like this. It may be that the variance across units is something you don't want to shrink towards the mean because you want to use that for follow-up or something. And so it as part of the the set of posteriors you conclude with would be ones that do very little partial pooling at

all. I think Andrew you call this the secret weapon or something like this, or you always compare the shrinkage estimates to the fixed effects estimates and that I think that's just a s a standard part of the workflow or it should be because even if you're gonna use the the partially pooled the shrinkage estimates, you learn so much about the data and the phenomenon by seeing where the shrinkage has happened. And if it if it's happening in places where you have other reasons to suspect that the detectability is of a particular type, then that that that'll start your mind going in ways that maybe you don't know where it'll go now. But I think is that thinking of of the workflow And the models you explore as a way of asking questions about this complex phenomenon that is just not possible to study in detailed ways with measurements because you there are places if I understood your question right, there there are some sites where you can get you can get i estimates of detectability and and recoverability or whatever word we want to use, but there are other sites where there's nothing, right? And And so your zero inflation is is haunted by this measurement problem. And and a a a set of models, a suite of models, maybe that's a better word, will help you explore how sensitive inference is to that problem. If it turns out to be really sensitive to it, then that's a headline a result of your work. And it's not a bad answer because it to the extent that people want to take these estimates that you're delivering and develop policy around it or something of that kind if there's an applied issue, then saying that this is kind of a a very difficult measurement problem for this reason, maybe that suddenly conjures effort to do measurement better in the future. So it's you could you're an active participant as an applied statistician in how data gets collected in the future, but you you need the the suite of models to illustrate to the problem to the psychologically healthy people who don't do statistics for a living. But but anyway, so those that's way I would approach it as a workflow and try to discover the issue and and sense it out. sorry, I've got a cat who's everyone's speaker-1: The daddy's here. Famous one. speaker-2: waiting for my cat. But anyway, but it sounds like a great problem. And yeah, we could follow up on it more later. You're not so far from for me here at Leipzig, I guess. But you're my neighbor in geographically speaking. But yeah that's that's the workflow po place that I would go with it is to think of it as trying to look at how sensitive these practical estimates are going to be to things you can't directly measure is is the way I'd look at it. speaker-4: Okay, so you understand the problem correctly and thank you very much. it's very valuable and it's a lot of information for me. So speaker-1: Yeah. Yeah. Anything you wanna you wanna add, Dorothe? speaker-4: Hm No, not really. no, I just well, I am honored that I c could talk to you

with about this problem. No, I don't have anything else. speaker-1: Yeah. Feel free to follow up with me anyways afterwards, you know, we're in contact. So but I think before you leave, Aki, you wanted to add something? speaker-4: Yeah, like the this bad measurements or missing measurements. So I'm right now working on cancer survival analysis and one of the measurements is mitotic count. So they take biopsy and then they look through microscope and count how many cell mitoses are going on. So these cell divisions. And for some reason they have used the the kind of the visual field in microscope and then how many mitosis you see in that field. Microscopes did get better. So the visual field did get bigger and bigger. And they still kept counting how many. cell divisions you see in the field. Only very lately then someone in some conference said that hey, actually we should standarise this and now it's five millimeter 5 square millimeters. But the problem is that we are now analysing the data where like 80 percentage of the observations are still with this field of view. From different years. And because it's from different years, different sites, different countries, we have no idea what the measurement is. Like in how big field how many mitosis. So it's it's very similar to your problem that you know it's zero, but you don't know why. Okay. So thank you. speaker-1: Fantastic. Yeah. Yeah. Thank you so much, Torta. I wanna really wanna thank you for taking the time and also having the the courage to do that. It was not an easy exercise. It's like literally you it was your first podcast. speaker-4: You know, I will start breathing after I will leave the room. speaker-1: No, yeah. That was that was great of of you to to do that. Like literally your first podcast on a topic you're you're learning and presenting it live in a very short time frame to great minds like Andrew, Akin, Richard. This is it takes a lot of courage. So yeah, thank you. Thank thank you so much for doing that and and we'll keep in touch. Okay. speaker-4: Bye bye. speaker-1: Bye. Awesome. Folks, so yeah, thank you so much for doing that. I think that was a great segment. Listeners, let me know how you like that. If you like that, I might I might do that a bit more. It takes a bit more of preparation for me in anticipation, mainly organization, which is not my strong suit. But if that's if that's worth to you, I'll I'll do it. I think it's a fun it's a fun segment. Aki actually I wanted to ask you about something that I think is related, like kind of a subtopic of what Dorota asked about. And it's this idea of trying to detect assumption dominance. I'm really interested in that because I think it's also very concrete in in a lot of models. At least I work on but I'm sure a lot of people because there is always that issue somewhere you have, you know, some clusters or some part of the domain of the model where you don't have enough data to inform the model. And so do you what do you

recommend to do to detect assumption dominance here? Like the the tools I'm thinking about is sensitivity and calibration, mainly, but yeah, I'm I'm really curious to hear what you what you have to say on that. speaker-4: yeah, at least the sensitivity analysis that how much your inference on quantity of interest would change with different assumptions and then like the already Briar sensitivity analysis is one of that. Richard mentioned said that my package, but it's Noah Gallionen who wrote the package. And Noah is now continuing also on this dataset sensitivity. Like he started to work in ecological problems where then the models are combining data from different sources, and then the interest is that how much these different data sources are actually affecting the inference. Because so far they are just combine them everything but how to then to computationally efficient sensitivity analysis also which data. But then it's the same whatever assumption is made or or any kind of this kind of suspect that there's something missing can be changed to be assumption and then we can just try. Richard would just simulate every time, change the this assumption and simulate but then if we don't want to simulate too many times I guess this kind of the important sampling based approaches would be possibility. To look at the sensitivity and like local sensitivity, kind of the gradient of what if we change something. speaker-1: Yeah. Yeah, I have to say also. speaker-2: I I just said that sounds great. I mean that's that's a very routine part of workflow and having a a more principled and efficient way to do it, that's that's really where the money is. speaker-3: Well if you imagine like just like what it would look like if it's all working smoothly, imagine you have some display of your data, like a scatter plot or maybe a grid of plots. And then but let's say just a scatter plot of your data and your model might have some number of parameters in it. You could have this scatter plot a bunch of graphs that look very similar. It's the same scatter plot over and over again, one for each parameter. The same scatter plot. But for each one, you color the points based on how influential they are for inference for this particular parameter. And then you'll find, well, some parameters are very sensitive. Yeah, cer certain data points are really telling you about this parameter, certain data tell telling you about that parameter, perhaps a transformation can allow you to kind of more clearly isolate what what matters. And then prior, that's just data too. So that's but yeah, you'll have it's something we don't fully understand. I mean we do discuss this in the book, for sure. and but like so let's see, like if you have well, sir, first simplicity, suppose you have a model with two parameters and you just have two data like You have parameters theta one, theta two, and theta one's estimated from a bunch of data, theta two is estimated from a bunch of other data. And you have a prior, a joint prior on theta one and theta two. Well, what's the influence of the prior? Well, in this case it's a separate if if your priors are independent for the two parameters, the speaker-4: And speaker-3: data are independent, then it's a separable model, meaning that the equivalent sample size of the prior would be different for theta one than for theta two. In fact, you could imagine you could have a situation where you have a very strong prior for one parameter and a very weak prior for another parameter, and that that would map onto influence. But if theta one and theta two are if the two parameters are entangled in the likelihood, then getting inference about one parameter will give you automatically give inference about the other. So for example, if you have a regression. And the eck with one predictor and the predictor's not centered at zero, that if you get the there's a mapping between the slope of the regression line and the intercept. If the if the center is not at zero, then the intercept will be at the extreme. So if your data are positive, when your slope goes up, your intercept will go down. So now imagine you have a strong prior on the slope, but no no information on the intercept. Well, it does that will still give you that actually becomes informative about the intercept through this correlation or or vice versa. So it's not always intuitive. And I I think that's one reason why we have to think about ways of displaying and and understanding it. speaker-1: One of the recurring themes I got in the the applications to the contest was how to how to communicate the results. Like there was a lot of submissions about that. and in particular there was one from a listener who's called Philippe Calvet. And basically his question was for people trying to bring Bayesian workflow into regulated industry, so something like credit risk where the incumbent is a deterministic score everyone trusts. What actually works to get committees and regulators to accept a possible distribution instead of a single number, even though it can be a comforting number? Is that a problem you you had in your in your own workflows? How do you recommend dealing with that? speaker-2: is this to me? Should I answer this? I I think speaker-1: Go ahead, yeah, I thought you would chime in. speaker-2: I have opinions here. I know my colleagues do too. But I think I think for for pure inference projects, which is different than the problem you're giving me, but it's a place to start. So if like a pure inference project, I often think that posterior distributions are not the right summary. Or at least the posterior predictive is a better summary. It's is something not not the posterior marginal distribution of some parameter. It's like it I I think of parameters as these gears in the machine and they they c they cooperate to produce predictions and scenarios. And so from a decision maker's perspective on the end, someone who's not a statistician,

interpreting a parameter is very difficult because the scale may be all wrong and You know, you've you've learned that the posterior distribution has a has a mean of 1.2. Well, what is what does that mean on the outcome scale? You've got some link function and it all gets transformed. So having summaries on the outcome scale, some posterior predictive, and it could be a posterior predictive contrast for causal effects or something, it would be my default that I recommend to people. And this is what I present to my course increasingly often. but for what you're saying, it sounds like you know, you're you're making a presentation to people who are used to making decisions based upon just numbers, just like this is the estimate and there's no uncertainty, right? It's just a deterministic score. In that case, I'd I'd go all the way into decision analysis probably and talk about costs and benefits of the different decisions that they could make based upon the existing information. I could say like and that 'cause that's that's a world that in my experience, that's a world that those folks live in. They they think in terms of the costs and benefits of taking action. And often postponing action is not possible. So some decisions going to be made, whatever information you give them, so why not meet them where they live and talk about consequences? Yeah. And that's what I would often do. I come from I didn't mention this at the beginning, but I come from a military family. I'm the only member of my family to go to college. And I'm surrounded in the the McElreath clan is a bunch of professional military people. So I l the when I talk to them, that's the way that they feel about everything, this stuff. They're like, well, we have to do stuff like all the time. And everything is like lives saved, lives lost. And that's that's where these people live. And whereas, you know, scientists like me, like I study human evolution, like, you know, no one's gonna live or die based upon the estimate of how old a Neanderthal is. But so yeah, your friend did or your colleague Id who lives in the real world. I think yeah, the costs and benefit of s some utility function, and you integrate over the posterior so Bayes, I wanna say Bayes still has huge advantages there. Bayesian decision theory is the business. And right, our our colleague Christian Robert, this is his expertise, isn't it, Andrew? And and he's got that great book. on Bayes that has a bunch of decision theory stuff in it. And we that's what I would recommend. speaker-3: We have yeah, chapter nine of BDA three, we have some dis real world decision theory examples that I really like those. I it was hard to find because decision theory textbooks typically have fake examples. So we we put in some effort for that. I will say there there is an opposite mistake that people make, which is people who are doing scientific inference who are like trying to be more hard nosed than they really are. And they're like, Well, you want to do all this Bayesian stuff, but we want to make decisions. We're decision makers. You have to be realistic. You know, they need to know what drug to approve. We need to have P values because we have to make because we're always saying you have to express your uncertainty. So don't say that Because your summary statistic is statistically significant, that it's real or that's not real, you know, like that. We're always talking about uncertainty. You get these people who are not actually decision makers. They're they're academics who will kind of scold us and say they're trying to help people who have to make real decisions, and we're not helping by emphasizing uncertainty. I, as you could tell by how I just spoke, I don't agree with that attitude. I think that. Decision first, like decision makers don't make decisions entirely based on a statistical summary. that's true. It's not true that if you do an experiment and you get a result that's two standard errors away from zero, that a good journal will necessarily publish it. You have to have a lot of theory and explanation. It's not true that a drug will be approved just because blah blah blah. So I do feel that our job as statisticians, unless we're specifically asked to give a decision recommendation, I think our job typically is not to make a decision for somebody. Our job is to give people a sense of the uncertainty and also a sense of what what more could be learned that could change things. So one thing I like to say is a probability is not Just a number. It's part of an it's exists in a network of conditional statements. and the example I'll always love to give is either you're flipping a coin and it's head or tails, or the world's greatest boxer is fighting the world's greatest wrestler to the death and who wins. Well, I don't know. It's kind of 50 50. How are those problems different? One is that like there's really nothing I can do with the coin. before flipping it that will give me information about that it's 50-50 no matter what. That's a very hard 50-50. There's nothing much I can do about it. The boxer and wrestler, I could learn a lot by having practice fights and things like that. I could gather data. So often one of the roles of the statistician is to say it's 50-50 or whatever, but another role is to say what data could be gathered that could could change things. And that can upset people. I mean people don't like it. I I people were annoyed at at me for election forecasts when we said the election was that the presidential election was roughly equally likely to go either way. They they felt that we're supplying no information and we had to explain that saying that it's roughly equally likely to go either way is information because not every election is so close. But it's it's not intuitive, right? People often will look at the outcome and not look at the process, which I guess is kind of big point of Bayesian workflow that the process is is part of it too. speaker-1:

Mm-hmm. Yeah. Yeah, one hundred percent. And in in my experience also it depends a lot on who the decision makers are in front of you. like I've definitely worked with complicated decision makers who didn't like nuances, but I've also worked with decision makers who really were able to think in scenarios and probabilistic thinking mostly, and these make my job the the easiest because Usually the way I report so to answer also to Philip, I usually never report just one number, even if they ask me for that I'm like, okay, I forgot. I always report at least three scenarios and I and I frame them as scenarios and I take them in the different distribution of the posterior distribution different point. And usually have like a you know, median scenario, optimistic one and pessimistic one, and that's usually how I report these kind of analysis and then that usually generates a lot of discussion about okay so how can we make the median scenario actually become the the optimistic scenario actually the median one and and that relates to what you were talking about Andrew about okay what would it take? What would we need to get to that scenario? So yeah in my experience what will maximize your ability to convey the fact that you don't want to give fake certainty by just reporting one number is to report scenarios and not talk about distributions at all. Like that nobody cares about that's my posterior distribution. I use standard PyMC2Kitnet. Like nobody cares about that unless the the unless they ask you about it. Then then it's great. You've you've got a great person in front of you. But usually it's just giving the results. That works quite well. Aki Do you do you have anything to chime in on that or shall I get to another topic? speaker-4: At least you should remind me about the question. speaker-1: Yeah, that was mainly about how do you how do you communicate mm uncertainty to decision makers. That was mostly that was the gist of it. And I think the cat really likes that conversation because still here. Yeah, go ahead. speaker-4: Yeah, I guess the the Yeah, the one one case was the this like first time I was making these cancer survival analysis and then like their the important part was also that it had to be the like the model the prediction result had to be something that can be printed. Like I was told like it was like about ten years ago that for cancer doctors it has to be printed on a paper. So it was these kind of the maps where you can choose whether there's been rupture, what is the location of the tumor, and then you read from X and Y axis the tumor size and methodic count and then you see the color telling the probability. And this I was told that it was success that but in a way that like also like simplifying there that it's easy to read that there were only like five levels of these probabilities told and I'm quite sure then that these cancer doctors were also including other information they had and like they used it for Decision making

or providing recommendation for a patient that whether they should then start eating medication which has also then nasty side effects. So at least the important part is like that the result needs to be somehow easy to use. speaker-2: Yeah, yeah. There's tons to say about this, but I know you have more questions, Alex, but this this is this is such a huge area that's it's sort of underdeveloped sometimes. Yeah. speaker-3: No. speaker-1: I mean if you if you really wanna say something about that, speaker-2: I'm curious what your next question is. All these questions are are speaker-1: Okay. Yeah. Yeah, so you guys still have time for one more question before ask re chart the last two questions or does somebody need to to drop very soon? speaker-2: No, th I have food being delivered, but I think it'll be delivered after we're done. speaker-1: Okay, cool. Cool. Yeah. And I mean and and the cat can do that for probably speaker-2: Yeah, the cat can check the door and he sp speaks German, it's perfect. speaker-1: Yeah, I mean I can see that. Yeah, yeah. He's be green being greeting people from the window. So yeah. so yeah, I have actually a fun colo mix of causal inference and modeling question that I got from another listener, Ian Cosley. and yeah, I really like that one. It's a bit of a you know, it's like it's a thinker, but I think you you guys will have interesting things to say. So so here it is. Imagine you send out a promotional mailer. And what you really wanna know is whether redeeming the coupon causes people to spend more. So the pro the one of the problem is that only a tiny slice of people redeem, so you cannot just compare redeemers to everyone else, also because they are different kinds of people. So usually the way the the trick that's used for this kind of analysis is just to lean on who received the mailer as a stand-in. Since that part is random. Mm-hmm. But this trick only works if receiving the mailer affects spending only through redemption. Which is probably false here, because just seeing the offer in the mailbox might nudge someone to buy. So that's kind of a contrived example, but I think it it speaks to a lot of different analyses, especially causal analysis, when you want to understand the effect of an intervention, basically. And so when you when you know the question is when you know your clean liter shortcut like that is leaking, can you build the leak into the model and let it tell you how much that leak could be distorting your answer instead of just pretending the leak is not there? speaker-4: Yeah. speaker-3: People have done that. the principal stratification model. So there's some I think like Avi Feller and some other people have done some Bayesian principal stratification models in Stan. And the idea is that you would characterize people that there's an a latent variable. Well, it's observed for the people who got the mailer. and not re avail not observe for the others. And it's whether you would have well I can't remember how it works, but like the people who the qu the latent variable is if you receive the stage one treatment, would you do the stage two treatment? And for the people who did receive the stage one treatment, you you know that. But the people who didn't, you don't know that. but you can form a model and you can do inference on that. And you'd want to use pre treatment predictors. You'd want to use characteristics of the people in the study. that the, you know, their their age and where they live and whatever spending patterns they have in the past. And then you can do that you should be able to do that model. And that will then you're at your es you can then from that you can estimate the effect of the you can c you can You can compare the people who would or would not do the stage two treatment and you can estimate what would happen if someone who didn't do it were to do it. But basically you have to you'd model the whole process. I I I haven't done this myself, but I know people have done it. It seems like a a very natural Bayesian problem. And I I think that standard Standard solutions such as instrumental variables would correspond to special cases of this model, assuming various things are zero or with flat priors and various and usually standard procedures usually com correspond to some mix of some flat priors and some parameters and priors with spikes at zero on others. speaker-1: Hm. Okay. Yeah. And so Richard, I'm I'm I'm sure you you wanna say something on that. speaker-2: Well no that sounds reasonable to me. I recognize this as the like per protocol or intent to treat kind of problem and and there's I mean that's a big literature and there's what Andrew says sounds like it's very sensible there. You you try to do the best you can, but you you gotta be careful about what you're adding because now you have it's this downstream from treatment problem. But people have written a lot about this. But this is the yeah, the per protocol intent to treat. sort of issue. If you want to figure out mechanism how the treatment's really working, it's it's hard even if the treatment's randomized. But there are ways to do it, as Andrew has discussed, and that's that's the way I approach it. Yeah. speaker-1: Awesome. Yeah, I love these problems. Like I think the these kind of hard causal inference problems are among the the most interesting ones because also that's where you get to to do the the most customized models and and that's these are the fun ones where you have to these are kind of Lego bricks that you you wanna you wanna add together. So I love working on these kind of of models. actually so we're I'm gonna start winding this down here, please start getting late for for Aki and Richard. I still have so many questions for you guys, but maybe so one is is there is there something you wanted to mention here that we didn't get to today? speaker-2: Well, that's a very open question. Yeah. We didn't speaker-1: Yeah, maybe something

that's not speaker-2: talk about we didn't talk about AI and workflow, and I think that's a very big and interesting topic. maybe w you'll have to have us back and we can talk about that. But that's I think that's a big area and it's very dynamic. and it's just here. Yeah. And Anyway, but you've you've we don't have enough time before my food is delivered, to discuss the speaker-1: I know, I know. I am very frustrated and and I definitely wanna get your thoughts on that and also what what you're seeing, how you're using that. I know I've been also of course doing a lot of that myself, working on on some major skills with some some people like Stefan Radf, I don't know if you know, but they've been on the show and and a lot of of great people doing that. So so yeah, I'm very interested in that also and seeing how it goes. speaker-3: I wouldn't get Jessica, Aki and Bob on together and to watch them scream at each other. but speaker-1: Sounds like fun. Yeah. Let's organize that. speaker-2: I'll bring popcorn. That would be great. speaker-1: Yeah. And Richard, I need to to have you anyways like for a solo episode on the show. Well we'll we'll dive on these on these questions. and and Aki, maybe do you wanna talk a bit about that? Because I know you guys do a lot of that at at Alto also a lot. So speaker-4: What like AI? speaker-1: Yeah, what I so I know you're working on that a lot with your group. Maybe like a no like a more pointed question would be, yeah, what what's on your mind for the coming month? You know, what are the things you're looking forward to that you're working on, that you're curious about? speaker-4: so or I'll like this go also back that like we've been working on the book like six years and it has a lot of also like in the case studies these software packages and diagnostics that I've been developing and this work continues and like it was just during the spring in the last moments adding like the corrected loop it plot approaches there and they're what's not in the book but will be then in case studies is now the blue package having these diagnostics when we can trust the unsynthetic quantification in blue comparison and These kind of things all the time coming more and it's kind of bit annoying that the book is actually printed. I can't edit the printed versions, even if we can like the electronic version can be updated and case studies can update it and we can add additional case studies. And of course like this is also connected to question about the AI that we know agents work best when they have tools so that the workflow is not just based on large language model but actually calling tools and tools that we trust. And then I think that that that's it's going to keep me busy for many years. like AI is not replacing me. I'm just then building tools for AI. speaker-1: Yeah. Mm-hmm. Yeah, I have to say what what you guys are doing at at Alto is super exciting. I will refer people to some episodes I've done recently with Luit Luigi Asherby, Jonas Aruda, Stefan Radef, like a lot of people who've been working on that. Osvaldo Martin probably needs to be back on the show. I know he's been doing a lot of Amazing stuff on the on the RV side. I definitely recommend people using RVs to check out the new version 1.0. there is so many new things in there, especially calibration plots. I love them, lose the use them all the time. and the the P sense also the prior sensitivity features, extremely useful to do the kind of things we've been talking about a bit earlier in the show, but in a more automated and an automated way with with guard drives. so yeah, all all this research is extremely exciting and speaker-4: The house folder has been great for putting things on Python side. speaker-1: Yeah. Mm-hmm. Yeah, yeah. Yeah. So so definitely we we we need probably to to do another another episode about that. Andrew, anything you wanna add on that? speaker-3: I'm wondering because should Aki, do you think we should have a web page like a version of our web page in Python? Because a lot of people are really okay. We we went through because i it's even I mean, I think it's kind of funny where you draw the line because in some sense our Bayesian workflow isn't limited to Stan either. But I think I feel I might be wrong. I think people recognize think even people who don't use Stan can think of it as a kind of universal language. So if you're using if you're using some hyper efficient parallel processing probabilistic programming language, there's still a logic to having the code in Stan and you should be able to to do that. but yeah, I think one of speaker-4: But Oswaldo is working on the Python version and is speaker-3: Yeah. speaker-4: is using also like his work time for that, so he's getting even paid for that, so speaker-2: Right. speaker-1: Fantastic. Yeah. Damn. Yeah. Thank you as well though. always, always doing great great stuff. Very very helpful for everybody. fantastic. Well, guys, I'm I'm gonna I'm gonna let you go because it's it's been a long time. I know it's late for for you, I can Richard. So let's let's call this a show. We could do a four hour episode, so I need to I need to draw the line at some point. but Richard, before you go, last two questions ask everybody at the end of the episode. I can enter already. answer that but first question if you had unlimited time and resources, which problem would you try to solve? speaker-2: Ooh, unlimited time and resources. Which problem would I try to solve? speaker-1: Can be anything, doesn't have to be statistics related. speaker-2: Yeah. Well, I I'll make it statistics related just for the sake of the listeners. But I think the problem I would try to solve is to move forward on what I think of as the grand p project in statistics is to make it increasingly formalized in the sense of of mathematical proofs. that there's this sense in which, of course, there's this old tradition, Aki knows very well computer science, of

thinking about algorithms as proofs. but it hasn't been applied very much to the way we develop statistical workflow. And I think there's a a lot of of homologies there and thinking about methods or algorithms for deriving minimal assumptions that justify procedures and so that we can really list, you know, make make things axiomatic in a sense. And I think that's been extremely powerful in mathematics and applied mathematics, and it would be equally powerful in statistics, but it's really a project that only got going in around nineteen eighty, I think, in a serious way, even though the foundations have been there for a long time. So I think that's a really important thing. And and now that said, I don't think I'm gonna make any important contributions to that because I have a job and I do a bunch of administration and I can't but you asked that hypothetical. But I think that's a big like generate multi generation project. for statistics is the axi making things more axiomatic and and finding ways to derive licensing assumptions, minimal sets of licensing assumptions. okay, next question. speaker-1: Yeah, love that. yeah, so second question if you could have dinner with any great scientific mind, dead, alive or fictional, who would it be? speaker-2: that's a tough one. I think well being here in Germany and at any time I would think Emi Nutter, probably one of the greatest applied mathematicians of her generation made really incredible contributions to physics, and that's that's a person that, you know, was right there in the middle of of all these other people and made all those other great thinkers better. And so I'd yeah, really love to talk to Emmy Nutter. I think that would be great. speaker-1: Brilliant. Yeah. you're the first one to to answer that, so that's that's great. Yeah, yeah. Fantastic. Well, gentlemen, thank you so much. it's it's very early for me in the day, but I have to say this is one of the best ways to start Wednesdays. So it's amazing to have you here all three. You're always welcome back. Again, thank you so much for taking the time and being on this show. speaker-2: Yeah, thank you, Alex. See ya. speaker-1: This has been another episode of Learning Bayesian Statistics. Be sure to rate, review, and follow the show on your favorite podcatcher and visit learnbasetats.com for more resources about today's topics, as well as access to more episodes to help you reach true Bayesian state of mind. That's LearnBaseTats.com. Our theme music is GoodBayan by Beba Brinkman. Fit MCLas and Megaran. Check out his awesome work at bebabrinkman.com. I'm your host. Alexandora. You can follow me on Twitter at Alex underscore Andora like the country. You can support the show and unlock exclusive benefits by visiting patreon.com slash learnbase dancers. Thank you so much for listening and for your support. You're truly a speaker-0: Good basie and change your predictions after taking information. And if you're thinking I'll be less than amazing, let's adjust those expectations. Let me show you how to be a good daisy. And change calculations after taking fresh data. Those predictions that your brain is making. Let's get them on a solid foundation.