speaker-0: ⁓ My guests today are Bill Engels and Jesse Grabowski. And if those names sound familiar, it's because Bill wrote Pymesy's original Gaussian process submodule during a Google Summer of Code years ago. And Jesse is a returning guest who came on to talk the state space models and Kanman filters back in episode 124. This time They are both here to talk about Gaussian processes and specifically about PTGP, a brand new opinionated GP library built on top of PyTensor and PyMC, with a lot of help from Jesse's work on PyTensor's rewrites system. We get into what a kernel actually is, what length scale and amplitude really mean, why GPs are so expensive to fit exactly, and how PyTensor's rewrites are. Let PTGP stay fast without hiding the math from you. We also talk about HSGP, inducing points, and when to reach for which approximation. And since both of these guys build tools that people now use side by side with AI coding agents, we spent real time on that too. How to write a library and documentation for a world where an agent might be the one reading your code first. This is Learning Vision Statistics, episode 166, recorded July 9. twenty twenty six. Show you how to be a good basy and change your predictions after taking information. And if you think it now be less than amazing, let's adjust those expectations. What's a Bayesian is someone who cares about evidence. Welcome to Learning Basion Statistics, a podcast about patient inference, the methods, the projects, and the people who make it possible. I'm your host, Alex Andora. You can follow me on Twitter at Alex underscore Andora. like the country for any info about the show. Learnbase stats.com is Lab Place to be show notes, becoming a corporate sponsor, unlocking Bayesian merch, supporting the show on Patreon, everything is in there. That's LearnbasedSats.com. If you're interested in one-on-one mentorship, online courses, or statistical consulting, feel free to reach out and book a call at topmate.io slash Alex underscore and Dora. See you around, folks, and best Bayesian wishes to you all. B Langles, Jesse Grabofsky, welcome to Learning Patient Statistics. Yeah. And Jesse, welcome back actually. ⁓ I think you've been on the show.
speaker-1: Huh?
speaker-2: Yeah. I improved my background. There were complaints about last time I had like a dirty sock stuck in Shanghai. My internet connection was terrible. ⁓ so here
speaker-0: Yeah yeah that's true.
speaker-2: Yeah. It was like some behind the scenes that my connection was so terrible, I think it took us three hours to record that and then your editor guy was just like, What are you doing?
speaker-0: Yeah, yeah, I remember now. Yeah, yeah, yeah. I mean, but state space models are worth it, so you know. ⁓ so that's fine. You know. ⁓ something. Yeah, yeah, exactly. but thanks again for joining. Live from Chicago. It's ⁓ starting to get late for you, so ⁓ thanks a lot. We we appreciate it with ⁓ with Bill. ⁓ Bill, you're live from Portland, Oregon. I remember the background. ⁓ from the time we we used to work together.
speaker-2: But thin's the juice is worth a squeeze.
speaker-0: ⁓ so ⁓ let's ⁓ let's talk about you actually because ⁓ it's your first time on the show, but finally it's ⁓ it's a long overdue episode. I managed to lure you in, of course, with both GP and Jesse. So
speaker-1: I begged and begged and begged and you finally let me come on. So thank you. Thank you. So yeah.
speaker-0: Bill is making me look good here because I was the one begging. So ⁓ but thanks ⁓ Bill for ⁓ for saving my feelings. yeah, actually can you ⁓ let's do your ⁓ your origin story, you know? ⁓ how did you come to what you're doing today? You know, you now you do a lot of base stats and you do a lot of Gaussian processes. ⁓ you're like one of the Yeah, you're like the GP guy from the Pime C team, but How did that happen? Like where where did you go wrong?
speaker-1: Where did I go wrong? Man. Yeah. So that's actually a great leadoff, because ⁓ I ran into this stuff for the first time a long, long time ago when I was an undergrad at the University of Oregon. And it wasn't in a stats class or anything, but I was I was working on a physics research project and it was with the LIGO experiment. and the task was detecting Gravitational waves from core collapse supernova. And they use Bayesian methods there. So that's how I got introduced to what Bayesian stuff was. I remember at that time I thought statistics was just like you have dots and then you draw a line through them. Mm-hmm. And it was kind of really boring. You just why don't you just draw the line? So that was very eye-opening. Yeah, so that was a A Bayesian inference problem kind of at the core. That was my first introduction to Bayesian stuff and then working on that is also what led me to Gaussian processes actually kind of early before a lot of other stats stuff, which is which is interesting, I guess. A weird way to find them.
speaker-0: Yeah. Yeah, so you started working first on on Gaussian processes?
speaker-1: Kind of early on I was just I I just really liked working on this project and got kind of stuck on it and it was more fun than physics homework. And ⁓ you know, eventually the problem was was really that you had to make a a surrogate model for these simulations of core collapse supernova and GPs are used for surrogate models, but I you know, it was like learning about, ⁓ what's maybe machine learning, what kind of statistics? It's gotta be some kind of regression thing and yeah, and then kinda just stumbled across Gaussian processes like that. And then ⁓ yeah, that's that's how I was introduced to them. So before I learned much about statistics or machine learning.
speaker-0: Yeah, okay. Well, okay, so how is it? But is it's not the easiest no entry door to the world of snacks.
speaker-1: It was very difficult. But yeah, I spent a lot of time. It was really interesting. I spent a lot of time learning about it and I just eventually thought this was a lot more interesting, well, than the physics stuff, physics homework at the time, you know, and and kind of decided to go that direction instead of physics. And so ⁓ took a few stats classes, took a machine learning class and went ⁓ graduated and then went to ⁓ Went to get a job and doing data stuff after that. Yeah. But I don't think I had it all figured out then. I just barely kinda understood GP's.
speaker-0: Wow, damn. Right. Yeah, yeah.
speaker-1: But it was interesting to learn about other stats after seeing those because, you know, everybody talks about the special case of G P is regression, a special case of GPs are neural networks, and a special case of, you know, time series, AR models, whatever. And so they they have connections all over the place. So it was ⁓ it's kind of fun to learn statistics from working backwards like that. Yeah. And also learning more about GPs at the same time. So they've always sort of been the little touchstone for For something I can come back to or a a way for me to understand something, thinking about kernels.
speaker-0: Yeah. Yeah, I mean and thankfully you did that because then I learned a lot of GP stuff thanks to you. ⁓ basically thanks to your production on the on the Binsey side, you know. Tutorials, you know, ⁓ of like all the things I I read all of that, ⁓ the code for the the sub package, because at the time it went you still had to read and write code. ⁓ yeah. Yeah. Incredible. Yeah, it was another world. and ⁓ and yeah, like I mean I learned so much from all the Pine C G P sub package that you wrote. So thanks Lord for doing that. ⁓ and I
speaker-1: Back in the dark ages. Thank you, thank you. I'm glad ⁓ glad people seem to like it. So Yeah. It was really fun to work on. So I appreciate that.
speaker-0: Yeah, I'm sure I'm sure a lot of people will be happy to hear about you today because I've had a lot of ⁓ of people asking me to have ⁓ to have you on the show to talk about GPs, so ⁓ you know, I'm sure a lot of people will be very happy ⁓ in the in the community. How did actually so you wrote the sub package for GP four point C
speaker-1: Cool, I'll do my best.
speaker-0: During ⁓ Google Summer code, is that correct?
speaker-1: That's right. Yeah. Yeah. ⁓ so yeah, Google Summer of Code was great, by the way. I ⁓ it was really nice to have a period of time to get paid a little bit over the summer. I this was during my I was doing a masters in statistics when I when I did that. And it was also really nice to ⁓ work on some code, like a code project, which I really missed while doing my masters because it's all like math on paper. And I missed coding things that, you know, did something and were used by people. So it was ⁓ it was a great time to do that. But ⁓ but yeah, the the reason I got interested in Py ⁓ C before that was that same research project because I needed a software that you could do it's really specific. You could auto-diff through a Fourier transform, you could set priors and do stuff like that, and it was really easy to kind of extend and PyMC fits the bill there and and Theano, right? They know was the back end of PyMC at the time and it had Fourier transforms in it. And I don't think other stuff, at least that I could find at the time, did. So and doing gradients through a Fourier transform and then a GP was ⁓ ab above my skill level, let's just say. So Yeah, it was like the perfect fit to to look at that. So that's kind of how I got interested in Py ⁓ C and it was easy to experiment a lot and then was doing summer of code, it was like, well maybe some of the stuff I could ⁓ you know, include it in Py ⁓ C and so it looked like I got a lot of work done really quickly, but a lot of it was kind of floating around on my computer already and it was just ⁓ a lot of cleaning it up and making it nice and making a good API for it. ⁓ but yeah, so that's that's how I kind of Submitted that to Pi ⁓ C.
speaker-0: I mean, you had to write the code anyways before, so you know, it's still ⁓ yeah, it's still your
speaker-1: Round, yeah. But I mean the having the mentors and working with like Chris Fonisbeck and the other folks that helped me out at the time was super, super helpful to 'cause that was the first time I'd, you know, written code that had to be like good, that was gonna be used by more than a couple of people and was like visible in public. And so, yeah, it was great to learn about how how to do that.
speaker-0: Yeah, yeah, no, I mean I'm I'm guessing it was nice and and yeah, Chris is is such a great mentor and teacher that it's just yeah, my each time I get to work with him I'm I'm super happy and excited, honestly. Yeah.
speaker-1: Yeah, just learned a ton, super encouraging and yeah, it was great.
speaker-0: Yeah, honestly. and so that's that's how you you started the the G P route on Prime C. ⁓ and then like did you do you have any let's say, you know, stories from the trenches in how you used GPs for clients for models in in real life, ⁓ and and maybe advice you can give to to listeners who are curious about them but haven't tried them out yet.
speaker-1: Yeah. ⁓ yeah, I've snuck them into a few models over the years. ⁓ working at so I worked at a kind of a series of I guess just straightforward data science jobs. They weren't, you know, doing Bayesian stuff in particular or or whatever. And yeah, there were always opportunities to use to use Bayesian statistics. ⁓ and then and then GPs can fit into that. Because one of the reasons I liked PyMC was that You know, there's kind of the split. There's sort of like the GP model as like a machine learning algorithm. And you can also put sort of embed GPs inside of a you know, like a hierarchical model, like something you would make in Pi ⁓ C. And ⁓ and that sort of second thing I thought was a really nice thing to do that was kind of underused because there's it's really nice for that time when you have, you know, just some function. And you need to sort of fit it and it's within this larger model. And maybe you don't really care about what the result of it is. Maybe you do, maybe you don't. And you need to sort of get rid of that variance or separate that variance from whatever other thing you're modeling. And GPs are great for that because otherwise you maybe you're using polynomials or you're using splines or something like that. And you know, you can't predict outside of the window that those are based over outside the training data and then you have to futz around with the order of it and how to set the priors and it's just a bit of a rabbit hole. And GPs make that really slick where you'll just have a scale on it and then a length scale parameter and two knobs to turn and set priors on and all that model selection stuff just kind of falls out automatically. And so that's sort of why I liked, you know, using GPs in PyMC and ⁓ It was also nice to have that functionality built in so that, you know, what happens a lot when you're, you know, working on a Bayesian model is that you have to sort of build it up stepwise, right? Like the Bayesian workflow ideas and you sort of need to quickly be able to try stuff out and like, well, maybe this thing's a bit of a function. I could just throw a GP at it and see how it does. And just making that cycle faster for dealing with a little nonlinear function in your model is just a really handy little thing to have in your toolkit and something that's quick to use. So you know, there were there's definitely been times at at work where you're just working on kind of a regression model and you wanna use Bayesian stuff because you don't want to have to explain p values to anybody and the model itself is something not quite off the shelf and so you can just whip up a little Bayesian model and sample it and, you know, read the results off in a pretty straightforward way and ⁓ Yeah, help people out.
speaker-0: Yeah. Yeah, and I mean I resonate a lot with that for a practical view of GPs. ⁓ I think one of the best ways I understood them is yeah, like if you have some kind of function ⁓ that you don't really know what the form, the mathematical form is. ⁓ that's a good use case for a GP. ⁓ yeah. And that's a very good, usually very good first approximation. And then sure if you care about the actual formula, then yeah, ⁓ for sure. It's gonna it's probably gonna be better. ⁓ if not if not from a prediction perspective, at least from ⁓ computation perspective. huh. ⁓ GPs tend to be ⁓ tend to be computationally expensive, at least the vanilla ones. but yeah. Like basically, okay, I have some form of nonlinear ⁓ relationship here. let's try a GP, you know. and that's usually a super super helpful ⁓ super helpful approximation of what's really going on.
speaker-1: Yeah, because that's kind of one way to build them up is they're a prior over unknown functions. And you pick a kernel and the kernel is kind of like, what what variety of unknown functions are we talking about here? And yeah, and ⁓ you go from there, right? And ⁓ there's a few ways to build GPs up GPs up from you know, things that make sense. I like that version, and then I also another one that kind of comes in handy is that a GP is The same thing as a hierarchical model where maybe you have some continuous covariate, right? Like ⁓ one of the examples that people have in PyMC a lot is the it's like batting averages. It's one of those top line PyMC examples. And you know, a bunch of baseball players. Yeah, exactly. There you go. And say say you want to consider age as a covariate. So all the players now aren't exchangeable. They all have an age. So, okay, what do you do with that? Well, that's just a hierarchical model, except instead of all the players just being kind of independent players, they now have this age relationship. And that's you can model that with a GP with a length scale. And so it's just a slight step up from a hierarchical model, too. So there's a lot of ways you can sort of find yourself wanting to use a GP.
speaker-0: Yeah. Yeah, and you can do you can do hierarchical GPs. ⁓ so that's that's definitely super fun. ⁓ there are some tutorials I've actually we've been working on that together ⁓ with BL where there is a two part HSGP tutorial where it culminates in ⁓ in an example where we show you how to ⁓ build a hierarchical HSGP in PyMC. So that's exactly these kind of ⁓ these kind of of cases where you would have for instance ⁓ hierarchical higher level GP on age and and then each baseball player in your example would be ⁓ would be their own GP but that's ren ⁓ I mean that's sampled from The population of G Ps that we have on the population of players, which is the first G P the higher level one. So
speaker-1: Yeah, yeah, right. You can can go GPs GP's all the way down. Yeah.
speaker-0: Yeah. This is super cool. And and now is HSGP, so Hilbert Space Gaution Processes. I'll link also to a few episodes I have on that and the webinars. I think when I think yeah, Juan Arduz came on the show to explain the math really behind that. ⁓ but basically in a nutshell you wanna talk a bit about HSGP build because ⁓ I feel like it's been really a game changer for practitioners, honestly.
speaker-1: Yeah, yeah. No, this was ⁓ it was from a paper Solon and Sarka and then Yeah. Ru RuTort male. I'll probably just butcher the names horribly. Yeah. And that that second paper though was like basically like, ⁓ here's this thing from this other paper, and it would be really handy to embed in a probabilistic programming language because you it's it's just such a good fit because
speaker-0: Yes, yeah, yeah, yeah.
speaker-1: So there's lots of GP approximations out there. And this particular one works it works well for cases that happen a lot when you're working in something like PyMC and you want to throw a GP at some unknown function. So it works really well for low-dimensional things that have like a low-dimensional input, 1D, 2D, maybe 3D, but don't go higher than that. And when you're kind of using the common stationary kernels like exponentiated quadratic and the Matern ones. And those are yeah, super common. And you can kind of build these like, you know, generalized additive model type models with them. And they work really well in there because the approximation is it turns it into a linear model. So it looks a lot like a polynomial model or a spline model where you have this set of basis functions And then the weights for each basis function are parameterized by the length scale parameter. And you end up writing something that looks like a linear model and it samples really nicely. ⁓ it's ⁓ it's an approximation that works well with MCMC. And so having that sort of ready to go and easy to use in in PyMC was super, super handy. Because yeah, GPs are slow. That is their big problem. Yeah.
speaker-0: Yeah yeah yeah for sure. ⁓ so I'll link to a bunch of these tutorials and episodes in the show notes folks. ⁓ feel free to look into that. I know we're ⁓ throwing a lot of technical terms here. ⁓ kernel, length scale, amplitude. We haven't defined any of them.
speaker-1: Yes, sorry.
speaker-0: I mean it that's fine. ⁓ we'll we'll probably do it. ⁓ we'll see f w how how much in the details will go. ⁓ depends on what you wanna talk about and and show for a PTGP. But first before we do that, because we're ⁓ twenty minutes into the recording and I haven't had asked any question to Jesse, which is ⁓ very rude from me. ⁓ even even for a French guy that's rude outside. ⁓ so ⁓ yeah, Jesse So you've been on the show episode one twenty four, it's in the show notes, folks, in the related episodes, I really recommend it because if you wanna hear about Jesse's very interesting background, ⁓ very diverse and ⁓ very interesting that's gonna be on that one. Today it's gonna be all technical from ⁓ Mr. Grobowski. But yeah, like ⁓ what's your experience with GPs, Jesse actually? ⁓ they like do you have worked a lot on on them or are you here just to learn?
speaker-2: I'm I'm the ⁓ I'm here for moral support. Actually there's so Bill started with this funny quip that everything's a special case of a GP. There's a nice coincidence that a common filter is a sequential GP solver. So in a special case where your covariance matrix is sort of blocked diagonal because you have these temporal
speaker-1: He's Yeah.
speaker-2: relationships, instead of trying to invert the thing all in one shot, like you're forced to do in a most GP land, you say, well, I'll just take it one block at a time. I'll do my one little bit of solution, and then I'll sort of march down the diagonal solving the thing. But the the complete solution is what we in time series land call a calm and smoother that sort of takes this two-sided look at the time series and gives you back a smooth ⁓ function based on your model. So direct GP, I'm maybe much less experienced, maybe, much less experienced than Bill. ⁓ but there's this very nice way in which the things that I work on and the things that Bill work on tie off. The other nice coincidence is that a lot of the math well, not a coincidence, as a result of the fact that these are doing the same fundamental linear algebra operations, a lot of the optimizations and w work I do in the background on PyTensor ended up really benefiting Bill's work as well as the stuff that I was doing in state space. So if we have time, maybe we'll talk a little bit about some of the ⁓ linear algebra assumptions, some of the new number back end things, things you can do in PyTensor with number now that you can't even do in number with number, which is really exciting.
speaker-1: Yeah.
speaker-0: Yeah, I think it's a good time now to to dive a bit more into PTGP so because that's why I invited you guys on the show. You have this new package out. PT is for PyTensor, or at least I think. And GP I'm pretty sure is for Gush and Processes. Yeah. Yeah. ⁓ damn. That's some good inference on my end. Yeah.
speaker-1: Yes. Creative name. Creative name.
speaker-0: Yeah yeah. but so yeah, well what's up about that? Like what's the idea, what's the ⁓ elevator pitch, why is that even helpful and interesting to people, Jesse? And I think it's linked to what you were exactly talking about.
speaker-1: Yeah. So GP's on PyTensor. ⁓
speaker-0: Wait, that's Bill. So for listeners.
speaker-2: So I mean it's good it's good for Bill to answer this because PTGP's his baby. Let me just say that if you go out into the world and you say, I want to fit a GP, as you know, coming as a more of a novice GP person, it is not at all clear what you're supposed to do. I mean, there's PyMC and I love PyMC, so you can do PyMC, but PyMC is really locked into this full exact GP Framework until the HSGP stuff came along. But if you want to do four or five six D inputs, HSGP is off the table. Exact inference becomes off the table very quickly. So there's GPjax, there's GPflow, there's GPyTorch, and they all kind of have like little idiosyncrasies. And I think what Bill accomplished with PTGP and what I've been hashtag blessed to be a part of is to build something that from first principles says like, We're going to be somewhat opinionated and we're going to be useful. That this is like the practitioner's package.
speaker-0: Yeah. Yeah, I love that. ⁓ I feel like it defines you also quite well, Jesse. Opinionated and useful. ⁓ I think it's fine. ⁓
speaker-2: There's that is the nicest thing anyone has ever said to me.
speaker-0: That cannot be true. Bill, so yeah, like ⁓ the ⁓ tell us ⁓ tell us more about PTGP in well, I I interrupted you so
speaker-1: No, I'm glad because that was a much better thing than I was gonna ramble about, so now I'll start rambling. But yeah, I think it's true. GPs are hard to use and this is my I guess I'll say my kind of semi controversial thing is that like if if if GPs were really, really fast, they would be everywhere. People would use them all the time, and a lot of other models would just be kind of useless. You know? Yeah. Which is a funny thing to say in the age of LLMs and deep learning especially. But ⁓ just because there are every other model is special cases of them and they're interpretable and they kind of sit right in this Venn diagram between statistics and machine learning. Like if you were gonna color color the middle in, it'd be GPs. And ⁓ you know, we talked about having GPs and Py ⁓ C where you sort of embed them within a larger probabilistic model. There's also
speaker-0: Mm-hmm.
speaker-1: probably more often the case where you just sort of have GP on its own, the GP model. Like I have some data set and I need to make a predictive model for it. And I really care about uncertainty in the predictions and I want to have some level of interpretability of like why this model is doing what it's doing. ⁓ then GPs are a great answer in that situation too. And that's a it's a pretty common thing, especially when you need a model that's accurate. And uncertainty matters a lot.
speaker-0: Yeah. Yeah. I completely agree with that. I think ⁓ I think there is like a bad reason for not using GPs in the sense that it's the same that it's a limitation of the universe is ⁓ computational limitations. And I think a b a good reason to not use GPs is if you have the actual mathematical formula for your phenomenon and you can derive it and you actually want it and that actually helps solve your problem better and faster. But it's like a very small use case, I would say. ⁓
speaker-1: I wanna.
speaker-2: Can I can I jump in and say I feel like we've been a little mean to GPs up to now to say that they're merely functional approximators or like they're black box in the way that a spline model is. Like a spline model, that's my black box example. I'm such a boomer. Like ⁓ neural network or something.
speaker-0: Yeah, have you just one further to you know like easy to grab.
speaker-1: Shapes. Yeah.
speaker-2: Have you ever seen that meme from ⁓ it's called like ⁓ I forget the name of the c it's a it's a superhero comic and the the guy's talking to his kid and there's like a fighter jet flying in the background and he's look at what they have to do to imitate even a fraction of our power. And I feel like this is GPs and it's like splines. People do splines, like you get basis functions, or people do like ⁓ these has support vector machines with like radio basis functions or people do nearest neighbors. And it's all boiling down to like, I have this notion that things that are close in the input space are close in the output space. And that's like fundamentally, this is what GPs give you. And it gives you this really rich toolbox to define a notion of closeness and to say that in this case, close means this, and in that case, close means that. And you can it's not merely right, like putting down, lying down basis functions and then saying, like, well, whatever the data says is what the data says. And I think maybe this is what Bill means when he says they live at this very beautiful intersection point between machine learning, which is like let the data speak, and traditional statistics, which is let's sit down and think really carefully about a problem and write down a structural model of how to solve it. And plus they have this Bayesian like je ne sais quoi, which really, you know, just pushes it over the top.
speaker-0: Mm-hmm. Yeah. Yeah, yeah, yeah. ⁓ completely agree. And actually they are a fun black box in a way. I don't think so. I think describing this stem describing them as black boxes is not very Yeah, it's not accurate as you were saying, because they are just I would say it's like ⁓ let's ⁓ maybe there is a good analogy. It's like they they are like ⁓ Shredding Shreddinger's models. You know, they are black box until you open the box. and then and then it's like perfectly transparent because once you once you observe data, they just become multivite normals, so it's like a perfectly known object. And they also are extremely interpretable, which is kind of like the opposite of being a black box, ⁓ because the kernel function is exactly what you just explained, Jesse, where it's like, okay, what's my notion of what does closeness means for my problem? That's gonna give you the answer to that question is gonna give you hints ⁓ as to which kernel functions to look at and select. And then you had the amplitude and you had the length scale. So these ones also are meaningly interpretable. I think, Bill, ⁓ you can you can ⁓ give that interpretation for us here.
speaker-1: Yeah, yeah, right. Sure. I mean there's your there's your definition of a kernel, right? Is you want to go, okay, I got this data point and take my X values, and I got this other data point and take my X values, and then the kernel function says it's a function of those two data points, and it tells you, okay, well, how similar should the Y's be here? Which is a really interpretable thing. And I think it's something that a lot of people kind of naturally when they think about modeling. That's kinda how they think about it. It's in a lot of ways it's easier to think about closeness in the input space and how that relates to the output than it is to think about like, well, if I increase this variable, does that increase or decrease the y variable and by how much? You know? Because then you have to s separate the effect of everything. And sometimes that's really nice and you want to separate the effect of everything. And sometimes there's a lot of interactions that you want to sort of have in the soup with you there. And GPs are when you know, kernels are really great for sort of thinking about like that. The it's almost like a spectrum. It's like you have these additive models with the terms and that separates everything out. And that's the easiest additive model to make. And then you can add interaction terms as you make your regression model better and better. And then GPs kind of work from the other direction where You have this kernel over all the inputs and everything's interacting. But then if you want to make your model more complicated, you split it out and you start looking for individual effects. And so both of these things kind of work at a spectrum and then meet at a middle. And then, you know, GPs, you can have the additive terms in the mean function. They call it the mean function when they're working that way. But it's really just, you know, all these things live on a spectrum of you have an additive regression model with. errors that aren't correlated and then maybe some of the errors start getting correlated with interaction terms and then a GPU over on the other side where it's like everything's correlated and and you have everything interacting and you can kind of you know it's it's nice to be able to work anywhere within that and start from one end or the other. So and kernels let you do that. And so that's kind of ⁓ what a what a kernel function is. And they're all composable. You can add them together. You can multiply them ⁓ there's like a little kernel algebra. They have to kind of follow a set of basic rules. and they're just functions that dis define, like I said, just to restate how similar given if these two X points for two different data points, you know, take these two, how similar are they? Well then how similar should the the Y's be? And that's kind of what the kernel's telling you very roughly.
speaker-0: Mm-hmm. Yeah. And length scale? So length scale is a terrible word for me. Like as a French person. Like in in French it would be a length scale. So what is a length scale, Bill? Yeah.
speaker-1: Yeah. Yeah, so a lot of kernels that we use a lot have a parameter in it called the length scale. ⁓ there's all kinds of kernels that don't have length scale parameters, they have other parameters. There's useful kernels that don't have any parameters, you just say they're similar and you go from there. ⁓ there's kernels over graphs, there's kernels over trees, there's kernels over all kinds of weird data things, but working in just sort of regular data where if x is five and another x is ten, the length scale parameter says, okay, is five and ten, they're five apart. Is are we going to think of that as similar or not? You know? And the length scale, what it basically does is scale the inputs. That's the simplest way to think about it. And you can do a little do a little algebra on a kernel function like the exponential quadratic, and you can see that the length scale just defines just divides each x by a scale. And you know you can divide x equals five and x equals ten, you know, by divide them both by five and now they're one and two. So now they're only one unit apart. And then you go, okay, well that's pretty similar if the length scale is five. But if your length scale is like a hundred, then five and ten are hardly different, right? That's basically the same number. ⁓ and if your length scale is like point zero zero one, then five and ten are really, really far apart. And those are Hardly the same thing at all. And so your y if you have X is five and another X is ten, then your Y values there shouldn't be related to each other at all. They they have n really nothing to do with each other. And so that's that's what a length scale is doing in in a lot of the common kernels like the ones from the Matern family and the exponentiated quadratic that you'll that you'll see if you look at the Prime C examples or or most examples with GPs. Yeah.
speaker-0: Mm-hmm. Yeah, and actually so you'll tell me how wrong that is, but usually when I teach GPs, a way I help students understand about length scale and amplitude is I use the first I think the most probably the most useful function of the whole PyMC package is the GP plot function that I think you authored during your G Soc, right? This plot that you get with the envelopes and like first the colors are amazing with GPs because they are so, you know, like uniform and continuous. It's just always extremely beautiful and pleasing.
speaker-1: Make pretty plots. That's that's honestly another reason.
speaker-0: They make amazing plots. Yeah, yeah. And you can use and you know you can use the best color palettes from Matplotlib, like you the VRDs and CDs and Magma and they work. Whereas usually if it's like if you've got categorical data as I have all the time, it's such boring palettes, you know, it's just like, ⁓ you know, it's just one discrete color after the other. It's like, ⁓ my god. GPs are great because continuous. So color palettes. That function is amazing. Because you can see really the GP if it's to D of course. And then so thanks to that, once you have that plot, basically the X axis is the length scale. And it's gonna like the length scale is basically gonna tell you how far do the relation go on the x-axis. Yes. ⁓ and it's like kind of a memory process. ⁓ the memory of the process, because like if it's a long memory, then the length scale is gonna be long. If it's a short memory, short length scale. And also sometimes the length scale can be ⁓ the memory can be spiky. It can be like if it's periodic, or actually every quarter you remember about something, you know. It's like ⁓ or like every year, you know, ⁓ shoot, I have to do my taxes, you know. So it's like that's like the the annual parody taxes, parody kernel, you know. so yeah, that's the length scale. And then y axis gives you the amplitude. The amplitude is basically a Scanning factor, and as you were saying, if you look at the most of the math of the kernels, the amplitude is just like, you know, that small factor just on the left of the kernel ⁓ formula. And that basically tells you how far can the functions go on the y axis. If it's a high amplitude, then well it's gonna it can go the GP can be very, very ⁓ capricious, let's say. If it's a small amplitude, it's like it's gonna be a very calm. Very calm T P, you know, like it's not gonna be
speaker-1: Yes. Yeah, I know. That's that's exactly right. It's the same thing because when you're thinking of those one D ones, you can think about a time series. You could say, okay, and and it's like a memory, like you're saying, right? So X is five hours and X is ten hours now, right? And if your length scale is really short, then it's like, well, it doesn't know anything. Like ten hours is nothing like five hours. So much has changed in that amount of time. So we don't know anything like what happens at five hours has nothing to do at ten hours. And so the same thing if X if the length scale is a hundred, then if you're talking on scales of hundreds of hours, then five and ten are basically the same time. And so then you're gonna have something smooth if you're if you're thinking about things in that scale. So it's it's the same that it's really funny you mentioned that function because just to go down the cause I was I just kind of hid that in there to help make plots for the documentation. Make pretty plots. But it has been useful and I use it all the time. Even just to plot one D stuff 'cause you it's like a nice shaved thing, but it it's funny, it has a horrible syntax and I just stuck it in there to be handy for plotting because I didn't want to have to like
speaker-2: Have you heard of have you heard of Hiram's Law?
speaker-1: No, what's that?
speaker-2: It's a law that that any any piece of your API, if you're a programmer, if you have anything, no matter how hidden it is, no matter how private it is, if it's out there for long enough, somebody will take it and make it an essential part of what they do and you must maintain it.
speaker-1: Yeah. ⁓
speaker-0: Yeah, eighteen. I don't know because I mean that that you know, that filters users, you know. Because if you make it easier to use then more users are gonna use it. And then you need to maintain it actually. Whereas if it's an Easter egg, you know like Yeah. If it's like, you know, animal style at In and Out Burger, you know, it's like just just the people who know knows it's there. You know, it's like it's fine, like you don't need too much maintenance. It's like you know like the the people who use it, that Easter egg, they you know, they know their stuff. So you could you could use that, you know.
speaker-1: Yeah. Just hidden deep in G P you've been
speaker-0: Exactly, yeah. C GPU tools. Yeah, but they use it all the time for like, you know, even for regressions and and stuff like that. It's amazing. Yeah.
speaker-1: Yeah. No, it's ⁓ I've tried to make it nice a couple of times and then it gets too complicated and I get stuck because I'm like, Well, this option or that option and now I have to actually make it good and then and then I get a little overwhelmed and have to do things that are whatever, do immediately, you know? Mm-hmm.
speaker-0: Yeah. JC, you wanted to add something.
speaker-2: I wanted to add that it's incredible that you've been on the West Coast for like two, three months and you're already preaching the gospel of in and out. Like you're really you're assimilating.
speaker-0: I do, I do. Yeah. So I mean so the fun story is that I already knew about In and Out because I was in Stanford in twenty twelve. And so that's where I discovered, you know, the truth. ⁓ so yeah, that was a fun that was a fun three month like I discovered in the same trip, In and Out, Jack in the Box, which is amazing when you're a student and you don't have money and you're hungry at two AM. ⁓ And and I discovered Netflix. That was the very beginning of Netflix streaming on your computer. And in France at the time it was the middle ages. We had like one big actor who of course didn't want that because they were selling forty Euros a month ⁓ ser premium service so that you would get that only on your TV and not even on demand. So so then when I arrived here, I was like, ⁓ my god. And I binge watched all the seasons of ⁓ twenty four, which was amazing. ⁓
speaker-1: Nice.
speaker-2: Yeah, I guess I guess the mini ted couldn't do streaming at that time. No, no, no, no. ⁓
speaker-0: But it's just you know, not not ⁓ it was the law, you know. I mean we like that in France, so
speaker-1: Nice. But in fact, I happen to live one mile away from the last blockbuster.
speaker-0: ⁓ yeah? ⁓ wow. Damn. And they're there sometime?
speaker-1: Yeah. I still haven't been in I have not been in because I do not have a VHS or D V D player, I will admit, but I need to. 'Cause I I mean, yeah. Streaming services are terrible now. I honestly would think it'd be nice to just go rent a movie. Yeah. Not like which serve ⁓ doesn't have it. Yeah, I have to unsubscribe from this one and resubscribe to that one. It's done.
speaker-0: Yeah, none's gone. The golden age is gone. Yeah, yeah, yeah. Now it's like now you actually need to pay like about one hundred bucks for your each month's food ticket.
speaker-1: Dumb terrible. I'd rather go just rent the one movie or series that you're gonna watch. Walk over and get a D V D and be a lot easier. Just the one. Return it.
speaker-0: So I actually who should ⁓ like you know, in in the in the biopic they'll do about you and Gaussian processes, ⁓ what b who should play your role, Bill?
speaker-1: Ha ha I didn't prep for that question.
speaker-0: I'll give you a few minutes then we'll we'll we'll come back to that. Yeah. ⁓
speaker-1: I ⁓ I always think about that. That's a
speaker-2: I see Ryan Ryan Reynolds, you know, you're like a lovable, like lovable loser type, right? And like
speaker-1: You find it.
speaker-2: They came out mean. I didn't mean for that to be mean. I meant for
speaker-1: I just heard lovable. That's all I heard. Yeah. I've gotten I've gotten Walmart Woody Harrelson one time. And that could be mean too, but that could also be positive because
speaker-0: Yeah, me too. Yeah.
speaker-2: Hear me out th hear me out though. It's Ryan Reynolds and the name of the movie is The Notebook. The Jupiter Notebook. Did you
speaker-1: Nice.
speaker-0: Turn out boo. ⁓ that's great. I I I'd buy I'd I'd go watch that, yeah for sure. Invite us to the to the premiere, please. ⁓
speaker-1: That'd be good. That'd be good. I'll ⁓ I'll prompt the AI to just generate that for the whole world. It'll be done in an hour. There you go.
speaker-0: So actually so to get back to get back to business, ⁓ why why did you use PyTensor for this new ⁓ this new package? You know, like what what is that adding what's the difference with the vanilla and HSGP sub module from PyMC? What would users even be interested in in that?
speaker-1: Because PyTensor is awesome. The end. No, but like it is though. It's awesome. And ⁓ One of the things so Pi PyTensor is like a you know, it's like a it's a very boutique hipster deep learning framework with auto diff. You know, nobody knows about it. It's not one of the big mainstream TensorFlow, Jack's, whatever, Torch, PyTorch, one of those ones. But it's like way better, actually. Because okay, Jesse should answer this one, but I'm gonna I'm gonna Bumble over a couple reasons why I think it's awesome is that you can if say, okay, say all those big companies decide to never support the big autodiff stuff again. And there's this new cool one that everybody starts using. And then everybody goes, ⁓ we gotta rewrite all of our code. And PyTensor, you can just wrap it. And now it's a back end. And PyTensor's still alive, right? So PyTensor is what Theano has kind of grown into over the years. And Theano is super old. Like how old is it? Thirty years? Twenty years? Thirty?
speaker-2: Two thousand two, two thousand three. Like the early commits aren't even on Git it predates GitHub, so the early commits are lost to time.
speaker-1: Amazing. Wow. Yeah. So that's it's and it's still around. It's still kicking because of this kind of ⁓ design choice where they had a C backend, but you could run other stuff. And so as long as you can kind of hook into it and run it from Python, you can run anything. So it's gonna be around. ⁓ and it also has this cool so okay, Jesse should take over at any moment I start saying things that are wrong.
speaker-0: Damn.
speaker-1: But like it makes a graph of your compute and it represents it in Python. Do you like I want to do one plus A plus B. It says, ⁓ there's A and there's a plus and there's a B. And then and if you do like A, I want to do A plus B divided by B, PyTensor can look at that and go, why don't you you just want to do A plus one? That's what that is. We don't have to actually divide B by B. So we'll scratch that out and your actual program is just A plus one. So it does these optimizations for you. So it kind has this little compiler step. And this graph it makes, you can work with it. So it's in Python. And it's not some deep C level complicated thing that you don't really have access to. It's just in Python. And that opens up a lot of things you can do with it too. So those are the two main things. And both of them are really handy for making ⁓ GPs.
speaker-0: Huh. But so that would be on the back end, right? Yeah. users would not see that. So
speaker-1: Well how unless they want to. Yeah, they might. Yeah.
speaker-0: ⁓ yeah, no, for sure. Then what do they get? Like, do you have actually examples already of of PTGP that you can show here or that we can put in in the show notes?
speaker-1: Yeah, ⁓ for sure. Yeah. We've got some stuff we could show. ⁓
speaker-0: That'd be great. So while you get while you will you get set up ⁓ you have that already? Just
speaker-2: Yeah, yeah, yeah. I'm all I'm all ready to go. I'm all ready to go. I'll see what I want.
speaker-0: Take it away and actually I'd I'd love to hear you, Jesse, about ⁓ the new stuff you were hinting about on PyTensor that made
speaker-2: I'm so glad you asked. What a what a coincidence.
speaker-1: Yes, there's a lot more cool stuff that PyTensor can do.
speaker-2: So ⁓ let me share my screen. Can you see this?
speaker-0: Awesome, yeah, yeah, yeah. Yeah yeah, so we have ⁓ Jupiter notebook, not yet ⁓ Bill's bio biopic, but ⁓ almost.
speaker-2: I need to I need to I need to correct
speaker-0: So if you're listening I I recommend switching to ⁓ switching to YouTube.
speaker-2: And like and subscribe on YouTube. That helps the show. True. The So I need to correct the record before I get into the Jupiter notebook because I made a I made a joke about who would play Bill and I got the actor wrong, right? It's Ryan Gosling is the heart throb with the abs who is in Jupy who is in
speaker-0: In the notebook.
speaker-2: Reynolds played. Which is Bill. I think it was a very good joke, but I got the actor wrong and I want the show to reflect that.
speaker-1: The girl?
speaker-0: Thank you. This is a show about ⁓ about accuracy, so thank you so much, yeah. I think you can
speaker-1: Yeah. The abs are the abs are spot on, so that part's
speaker-0: I think you can zoom in a bit actually, ⁓ Jesse.
speaker-2: Yeah. Yes, yes, yes. If I can, how does one work? Let's see. Command shift. Yeah. That's right. We gotta get the shift in there. That was what I was mucking up. So I'm going to jump to the the good stuff here, which is kind of how does this all work with a GP? So we've talked about GPs are math heavy, they're hard to work with ⁓ because they're expensive. Why are they expensive?
speaker-0: Come in place.
speaker-2: So I have on the screen kind of the two main equations for a GP. One defines what this kernel matrix is, and the kernel matrix is just a function of an input matrix X and it's pushed through some function. Okay. The function has to have special properties. Essentially it ensures that no matter what you give it, what comes out Hey Bruno. No matter what you give it, what comes out will be a valid covariance matrix. And then you're going to take this valid covariance matrix and you're gonna pass it into ⁓ the log p of a multivariate normal. And the log p of a multivariate normal has a really unfortunate property that you need to invert the covariance matrix. So this first term here for the listeners is y transpose covariance inverse y. So this inverse step is actually like O n cubed, a little bit less than O n cubed, like O N two point seven or whatever. It's really bad. And it's also quadratic blow up in memory. So if you have ten thousand data points, that means you need to store a ten thousand by ten thousand matrix in your computer. That's not so bad. But if you have a million data points, like you're kind of lost. So what can be done? This is I'll say one more thing, that the way that I've written this here with this actual k to the minus one is kind of the math way of thinking about it, where you say, I'm gonna actually invert this thing and then I'm gonna do standard matrix multiplications. ⁓ I say that evocatively because in a moment I'm going to make a pitch for why PyTensor's rewrite system is actually nice, even if it's on the back end. So what can be done here? If you're if you're sort of hip and you know numerical computation, you kind of know that k inverse y is a solve operation, and that you shouldn't do inverses, you should change this to be a solve. Right? So you can write this in code as solve ky. And then if you're really, really hip, you kind of know that, well, k is positive definite. And so I could do a Cholesky decomposition on K. I could store this. decomposed matrix. And then I could use a triangular solve or two triangular solves to get this y transpose k inverse y. And that's a little bit cheaper. And then all the way for free, we haven't talked about this log determinant of k. But once you have a Cholesky decomposed matrix, the Chalesky matrix is always lower triangular. And the determinant of a lower triangular matrix, as everybody of course knows, is just the product of the diagonal. So now you're not doing any kind of fancy Linear algebra, you're never calling out to Lopak because you already took the time to make this K, this Chelesky factor of K, you could just read off the diagonal and do standard math, ⁓ standard arithmetic. So GP libraries, what they'll concern themselves with is sitting down and thinking really hard about how do I write this thing out in the most computationally efficient way possible, and then stuffing it deep into the bowels of their library. And hoping that nobody ever looks at it because it's not going to be obvious at all if you just came from a machine learning textbook that showed you this equation, how it's actually going to be written once you go to the code. What's nice about PyTensor is what I'm going to do is I'm just going to write it exactly as it appears in the textbook. So I'm going to say that this quadratic term is just y Pt Linaugh inverse covariance y. And this is like absolutely verboten because now you're directly doing this O n cubed inverse. You fool. You've left money, you left cash on the table. And then log debt. I'll use the s log debt function which computes the log determinant of a matrix. And then I'll calculate exactly the formula that was shown above. And I'm eliding details about the X quad thing here and and and whatnot. And you could do that. Okay, now PyTensor has this suite of rewrites. Anytime you give it a program, it's going to look at the program and say, what do I know about the relationships between the objects in this program such that I can make it a little bit better, a little bit more stable, a little bit faster. In this case, ⁓ I have this counts linalg ops function, and it's going to tell us how many linear algebra operations are in here and what they are. And in this case, we end up with one s logdet function and one solve. So we didn't write a solve, but we got a solve. So already we've won. That's nice. Now what's new is that we have this assume function in PyTensor. And what assume does is it lets you tell PyTensor that certain objects in your program have certain algebraic properties that aren't that are not necessarily inferrable from just looking at the operations. In this case, we can tell it look, k is a covariance matrix, a covariance matrix is positive definite, and a covariance matrix is symmetric. It's going to remember those things and it's also going to reason logically about this over a lattice, like a logical lattice. And it's going to say, well, if I have a positive definite matrix and I take a dot product with another matrix, well, what do I know about that other matrix? Well, it's positive definite two. So that's going to stay. Is that true? Are you shaking your head? Did I get that wrong, Bill? Is the dot of two positive definite? It might be wrong. But the point is
speaker-1: No, I was I was you're good. Yeah.
speaker-2: Okay, okay. ⁓ the point is is if it knows something about another matrix, it'll propagate these properties forward. Here's a more simple example. I am a diagonal matrix, and I tell PyTensor you are diagonal, and I multiply it by some scalar. You know, I take i times four. Like while the zeros are all going to stay zero, the i is going to become a diagonal of fours, the result of that is just ⁓ a diagonal matrix again. And actually PyTensor knows. That, ⁓ if you're diagonal, I don't even need to look at the off diagonal elements. I'm just gonna take that diagonal, which could just be 10 numbers instead of 100 by 100, I'll do 10 multiplications instead of 100, I'll give you that 90% savings. So if I tell it that K is positive definite, and then I run the exact same ⁓ compile, then what I come out with is one Chalesky decomposition and one Chalesky solve. So you see the determinant is completely gone. That's become just a summation because we're doing the log of a product. So I just add the diagonal. And then we're doing one Chalesky solve to ⁓ actually do the ⁓ the solve got further specialized, is what I want to say. So more numerically stable, a little bit faster, really great. And everything like numerically is completely equivalent. This notebook will be in the show notes. We're gonna get it live on PyTensor. ⁓ examples gallery. So there's a bit more and it shows you kind of how you can extend it. And Bill actually did more specialized things. So he actually specialized the inverse function. We in PyTensor we specialized solve, but we didn't specialize inverse. So if you took the gradients of this, you would be un un unfortunately there would still be a matrix inverse left over and that's kind of a bummer. Bill said like, well let's just change that into another Chalesky solve. And we can reuse the Chalesky factor we already have laying around. So you don't even need to decompose it again. And then it makes it even better. You end up with four Chalesky solves and one decomposition. So very hackable, very extendable. That's kind of the power of PyTensor. and in this case, it gives us three things. It gives us numerical stability, it gives us readability and kind of maintainability. ⁓ and then the third, and I guess I could just have mixed it into the second, but three is a wonky number. It's approachable. Right. So again, if you're someone who's coming from a textbook and you want to understand how your code works, you go into the library, if you go into PTGP, the point is that you're going to see the closest thing to how it's written in the papers. And then we're going to lean really heavily on PyTensor to give us that same speed up that the g that the fellas in GP JAX and GPflow and get ⁓ from handcrafting these these functions really carefully.
speaker-1: Yeah, and that opens up a lot of stuff, right? Because a lot of these, like a Cholesky, there can be certain structure in the matrices and you want s different solves and and all of that's now on the table. But I want to just kind of highlight, Jesse, you went over something really quickly that I think is really cool, is that you wrote a a rewrite that replaces the inverse of A with a Cholesky solve when A is known positive definite and it was eight lines of code and you did not have to add it to PyTensor. It's just sitting in your notebook. And then now it got applied to the rewrites that PyTensor's doing. So and this calculation that you ran, is that using there's a Jax backend, there's a Numba backend, there's a C backend. Isn't there a what's the Mac thing? MLX? Right. Someone's finishing that up.
speaker-2: Yep, there's an MLS and there's also a PyTorch. So we have PyTorch. Nice and I mean
speaker-1: Yep. So you could run that in all of those just by changing a flag. you know, if whatever a certain a certain tensor library is not maintained anymore and there's a cool new one, you can use that one with PyTensor. And so if you're working on some library and you there's like, ⁓ there's a rewrite that's just specific to this thing I'm doing, you can just have it in your library without adding it to PyTensor. Or you can add it to PyTensor, one of the two. It doesn't whatever makes the most sense. And you know, you can just keep going with what you're doing without PRing it in and do that on the side and it's just it's a great library to work with.
speaker-0: And so thanks a lot, Jesse. Yeah, that's I think it's great to see it actually on the screen. ⁓ is there already a Python a doc ⁓ website for PTGP that we can link to? Awesome. Yeah. So I'll do that. Also link to to the examples. To make this even more concrete for listeners, if you if they wanna use PTGP, so they install it with Conda or ⁓ Peep. Does that depend on Pint C or is that just Pytensor?
speaker-2: I'm so glad you asked. Let's quickly look at what a PTGP model actually looks like.
speaker-0: Yeah, perfect. Yeah. I think it's great if we show basically yeah what what that looks like, how they would even write a ⁓ a P T G P model.
speaker-2: So the answer to your question is yes. PTGP depends on both or builds on top of both PyMC and PyTensor. And the decision was made very early on, like kind of right away, to say, what what is PTGP not? It is not a probabilistic programming language. And maybe Bill, you can talk about kind of your thinking there because it's I think interesting and important. But the point is that when you need a prior, When you need to simulate, when you need to sample, we're going to reach for a PyMC. When you need to do something with GPs, some specific GP logic, we're going to have functionality in PTGP that does what you need to do. So if you look ⁓ here, I have a model, a very simple model. ⁓ we're fitting on this motorcycle, the Silverman motorcycle data set, which is recording like the head velocity of a crash test dummy when ⁓ it impacts, yeah. So this is kind of the case that was being discussed in the introduction where there could be a physical model here. There's some Newtonian model of physics that describes how the evolution works, but we're gonna say we're not so interested in that. We're interested in understanding, maybe we're interested in like interpolating or we're interested in ⁓ extrapolating, or we're just interested in something else, and we want to kind of take out the physics piece of it and then study some other effect that comes in on top. And so the G P is gonna suck that up and then we can do whatever we So the actual model definition is just nine lines of code. We're going to define three priors for the length scale, for the amplitude, and for the observation noise. And those will just be normal PyMC priors. And then we're going to take the PyMC prior and we're going to multiply it by a P a PTGP MATERN52 kernel. And you can see kind of the the watchmaker's hand behind the design here. Because the PTGP kernels look very much like the PyMC kernels. So if you're coming from PyMC, this should all feel extremely familiar. And then
speaker-1: Like other libraries kernels like G Py or GPflow, I mean they all kinda use the same design for good reason. And yeah.
speaker-2: Absolutely. And then the the magic of PTGP is kind of this line here where you're going to specify the approximation that you want to use to estimate your GP. And I use this term approximation loosely because in this case our approximation is no approximation. We're going to fit the exact full GP. But w in the introduction I used the term opinionated and I used I described PTGP as the the kind of blue collar, you know, yeoman's practitioners ⁓ GPU library. Yeah, I I dressed for the occasion. I I put on a shirt just for that joke and I waited an hour for it.
speaker-1: ⁓ huh.
speaker-0: Well done. That's that's dedication.
speaker-2: The consequence of that is that what is provided are battle tested algorithms that scale up with good guidance on when you should use out approximation one versus approximation two versus exact. In this case, we have a very small data set. Everything's copacetic, so we can just use the exact thing and we can get an answer. The way you get an answer is you just call PT GP dot fit. Bada bing, bada boom. You're done?
speaker-0: Okay, and so that that fit stuff is what does it do, is that ⁓ L E thing or is that ⁓ C ⁓ C?
speaker-2: Bill, you want to talk about the philosophy of fitting?
speaker-1: Yeah, yeah, sure. ⁓ like I kind of said earlier, like there's the GPs that kind of make sense within a probabilistic programming language. So there you're gonna use MCMC, and then there's GPs that are sort of like a GP model, and this is models a GP. And that's kind of what's going on here. And in this case, we're doing map estimation, right? Because we've put priors on the length scale and the amplitude eta and the noise sigma. Like we've got those priors thanks to PyMC. That's already done. And so we're gonna run probably I think L L B F G S L F B or whatever, that one on ⁓ and fit is like a nice quick wrapper over the sort of default basic, just optimize this thing ⁓ using sci pies minimize. And ⁓ yeah, it's ⁓ you can kind of It's set up so that you can kinda get in there and tweak the optimizer because you can run into weird edge cases. Like Alex said, fitting GPs is hard and you gotta make it easy, but this is the easy code to just try it out and run it. Run the minimizer and go, you know, when you're in a a pretty straightforward situation.
speaker-0: Nice.
speaker-2: Although and although it is map, so for the kind of good Bayesians in the audience, you will know that map is sort of poo-pooed because it doesn't give you posteriors over the hyperparameters of the model. And what that often ends up hap what is what ends up happening is that you don't get your posterior predictive distribution, you're just point estimates all the way down. And so you took this beautiful probabilistic model that you could very well have fit in Perfect Bayesian form and just reduced it to Dare I say a frequentest result? ⁓
speaker-0: I'm gonna have to edit that part.
speaker-2: Note but just beep it.
speaker-1: We dropped that. ⁓
speaker-0: Yeah, I'll beep eat.
speaker-2: ⁓ Now what's what's about ⁓ Bit actually be like
speaker-0: It'd be like an editorial choice.
speaker-2: What's what's cool about GPs and what they have in common with state space models is that they're kind of Bayesian from first principles. And so even if you only have plane estimates for your parameters, you still get distributions out for your estimates. And so you can see here that we have plus or minus two standard deviations, everything's normal, so that works out to your ninety-six point whatever ⁓ percent HDIs. ⁓ and yeah, it's it's it's nice. You get a good result, it's fast. It's simple. We like
speaker-0: And so it's that Bill because I remember you explaining that to me ⁓ a few months ago or years ago where ⁓ you were like, ⁓ no, you'll you're still gonna get uncertainty estimation wave map. I was like, What? What water is mocking? ⁓ and you're like, No no, you get that because if I remember correctly, it's because it's ⁓ it's an empty normal, a multivite normal, and so thanks to that basically you get that. Uncertainty estimation. ⁓
speaker-1: Right. Yes, that's right. ⁓ because you're it's a Gaussian process, right? So you've got this kernel. The things you don't know are like the length scale and the amplitude. So those make your kernel, so that's a matrix, and then that matrix it's ⁓ it's in a big multivariate normal. So then you're drawing samples and then you have the uncertainty coming from that. And yeah, you are you are underestimating the uncertainty a little bit because There is actually uncertainty in those hyperparameters. But if you if you were to actually do it that way and use MCMC, you slow way down because you need to make a new matrix every time you change the hyperparameter and make a whole new matrix, and then do a whole new Cholesky and a whole new whatever to get one sample and repeat that every time. So by using map estimation, you don't have to do, you can do the Cholesky once on your map estimate and then draw lots of samples from the multivariate normal Instead of doing the Chelesky every time to just get one sample.
speaker-0: Yeah. Yeah, so it's like it's slightly I mean it's underestimating uncertainty for sure. In some cases it it will be slight, in some cases it will be high, so it will depend on the use case. and and yeah that's that's a choice. But what's the like is there a default way of sampling in PTGP? Is that is that the fit method or like people can just choose whatever they want?
speaker-2: So, and maybe Bill, you can disagree with me here, but essentially what is recommended is that you look at kind of the size of your data. And if you have very small data sets, we're talking hundreds of data points, then you can just go for the unapproximated. There should be no issue if you have a modern computer, you can easily solve it. There's no reason to cheat yourself of the full thing. As you scale up to something in the order of thousands to like five thousand, six thousand, there's this very nice Variational free energy approximation, which is ⁓ an induce it's a part of this inducing point family, and we're going to need to maybe talk a little bit about inducing points. I hope we have time. But this is kind of the most exact inducing point formula. Bill, if you agree with that, please.
speaker-1: Yeah. It's inducing points when you have larger data sets, larger, not large jest, and you also have a normal likelihood. You need that too. So there's a lot of these that's why GPs are hard. There's a lot of these like if this, if this, if this, then this is the best and and it's it's tricky to navigate that.
speaker-2: And if you have really huge data, just from say five to six thousand to infinity, then you're going to reach for an algorithm called ⁓ SSGP, S V G P I'm sorry, stochastic variational GPs, ⁓ because that can do mini batching and you can just let it rock with an atom optimizer tuned to whatever batch size your computer can handle, ⁓ so it sort of scales infinitely. ⁓ at least that's the promise.
speaker-1: Hopefully. Yeah. But no, it it scales up for sure. But yeah, yeah, yeah.
speaker-2: And there's there's more exotic and exciting tools coming. Things like you can take infinite data points and sort of project them onto a sphere somehow, and then that turns it into an O one problem. So no matter how big your data is, there's these exciting new approximation methods where the inducing points are picked in such a clever way that it scales constantly with your data. And then Everything's really, really fast.
speaker-1: Yeah, it's kinda trippy. The inducing points don't have to live in the space of your data. They can actually live in some other space. And in that other space, maybe it's more efficient to do linear algebra. So Yeah.
speaker-0: Yeah, I mean that makes sense. Like bas in inducing points the idea is like you try to summarize your your data set, right? So it's like I'm guessing does it have some familiarity with ⁓ PCA or stuff like that?
speaker-1: ⁓ Kinda. It's easier I kinda think of them as like spline knots. Sorry, Jesse, go ahead.
speaker-0: ⁓ right.
speaker-2: I was just I I was gonna push back on PCA just because PCA summarizes the column, the columnar dimension. So you have like a thousand features and I just wanna find so I wanna find ten features that summarizes it really well, but I still have as many rows as I have, right? Mm-hmm. The important thing about the the the most important thing about inducing points is that you say, I have too many rows. Like I don't need all this rows. I can I can get the same answer with many fewer rows. I just wanna carefully pick The synthetic rows I'm going to use to solve my problem.
speaker-0: So it's just like sampling. This is like polling.
speaker-1: Little. Yeah.
speaker-0: You have a full population and you're trying to find a the smallest representative sample unbiased of your population to learn the actual ⁓ election result.
speaker-1: It's exactly like that, yes. And the inducing points don't have to even be draws from that population. They can be c some kind of thing in that space or another. Yes. But they summarize it, yeah. They reduce the rows. Yeah.
speaker-0: They could be they could be like two voters or three voters like summarized and like d s you know, aggregated into each other. That'd be a weird poll, but I'd love to see that.
speaker-1: That's kind of what you think about if you're like, can I use inducing points here? You're like, is there kind of like a repetition in my data? Do I have lots of is my data dense in certain places? And you you want to keep it, you don't want to throw it out because you don't really know where it's dense, but you maybe have some sense that it's a bit too dense, you know? And it's like this function that Jesse's got showing on the screen is like the data's denser than what you would need to play connect the dots through the curve.
speaker-0: Curve.
speaker-2: I have a kind of more extreme example as well. This is the daily the daily monoloa data set, it's quite famous data set. And this is daily data going from the present all the way back to nineteen seventy. So we're talking about fifteen thousand data points. But if you see, if you zoom in, I mean the data is very highly structured. Right. And like you simply don't need all of this. Now I picked six hundred inducing points. So if you look at the bottom of the graph, and and for the viewers I have the monoloa data set which goes up into the right and has periodicity. It sort of wiggles up and down. And on the bottom I've plotted the the places where I've put down inducing points. And I put down a ton of inducing points. It looks extremely dense, but it's still like less than one tenth of the actual data. So even with this kind of extreme example, like I didn't bother to do anything smart, right? I just said, I've got this time series data. I'm just going to lay down a bunch of dots on the time series kind of evenly spaced. And then you get the same answer you would have gotten if you solved the whole thing. Because there's just too much there's too much re ⁓ repeating information in the data set.
speaker-0: Yeah, that data is amazing. I I'd love to get that data. I've never feeded I mean s feet or sample the G P on on that that data. It's incredible.
speaker-1: Yeah, it's a good example. Classic example.
speaker-0: Yeah. Yeah. And so all the all the methods to infer the GP parameters here that we've talked about, this is all already available in Pt GP, right? So feed, sample I'm guessing with ⁓ with classic nuts, and S V G P things like that.
speaker-1: Yeah, except for nuts because they're sort of we're we're still thinking about where to kind of split the line. Because that's that's a that's gonna be you're using that within PyMC and PTGP sits on Py ⁓ C It's a it's a fuzzy line. But yeah, there's gonna be no there's definitely not going to be like a different implementation of nuts that PTGP would use. You'd use NutPie or the sampler built into Py ⁓ C and maybe you'd use some PTGB code to construct the the GP itself or or not and then and then you'd use PyMC for the sampling for that one. But yeah, the the optimization stuff though is going to be in PTGP because one of the you know one of the goals I have is to make it easy to use, right? 'Cause ⁓ it's There's a lot of great GP libraries out there, and they kind of occupy different niches. And a lot of them implement, you know, the method from this paper or that paper, but there's not a lot of guidance on past the basics on when to use what and also how to fix issues when they come up. So often you can fit something and run into problems, or the fit will do this and that. is a sign of this pathology and this is what's wrong with your model and you need to fit it. And it's kind of like how how nuts is where you look at the trace and it says something about identifiability or something like that in your model. and it's a little simpler though because you're mostly doing optimization stuff. But ⁓ I just want to make we want to make sure that how to fix that, how to kind of cycle through models quickly because we all work under time constraints and we got to get stuff done. ⁓ so how all of that is ⁓ how how you can it's made with that in mind so that that's easy to do, you know? So you're not in that situation where you're like, well, that looks like a cool paper. Maybe I could try it. It'll take me a really long time to implement and then it doesn't work, and then is it because I did something wrong, or is it because this method is wrong and what do I do next? And ⁓ so sort of have the batteries included and the the help kind of included.
speaker-0: Yeah, yeah. Yeah, I love that. and G Jesse, you were were you done with the demonstration or did you have anything else to show?
speaker-2: Well, s we can do one more. I didn't want to ⁓
speaker-0: Yeah, yeah. Let's let's think it's great to so that people can see basically what they can do with that.
speaker-2: Hundred percent, yeah. So let me just point out so we're looking at the VFE approximation. This is the one that is for a medium amount of data points. I think I I pushed it pretty hard with this Monoloa data set. It's fifteen fifteen thousand data points. So it took about nine minutes on my my MacBook to fit, which is a bit longer than one would hope, but it it works. I I wanted to point out one thing Bill mentioned. That it's batteries included. So we saw this fit function, but everything is also totally decomposable. So there's this compile scipy objective function and it gives you back all of the bits that you need to then go ahead and call minimize yourself. So if you wish, you can ⁓ customize this as much as you want. If you love genetic algorithms, you know, you can go into the differential evolution thing in SciPy and you can give it your function and you can let it rock. Right. So ⁓ we do LBFGSB because this is a really good algorithm that's hard to beat, but we also expose all of the components. So if you're a user who's running into problems and you think that you can do a bit of tuning, that that's all available for you. Nothing ultimately is hidden from you.
speaker-1: And oftentimes, even in this situation, if it's if you're having something that can happen, kind of a failure mode, I guess, of this VFE approximation, is that you'll have problems optimizing both the inducing point locations, the noise, and the hyperparameters at the same time. Sometimes ⁓ and you'll you'll just see training. Do something weird. I forget, either sigma goes high or sigma goes low, because it'll be like there's no noise or there's all noise. And then but the fix there can be to first optimize the inducing point locations and alternate between that and the hyperparameters. And sometimes you want to hold sigma fixed for a while, optimize those other two things, and then let sigma start optimizing so it doesn't run off to one of those one of those bad spots. And so It's that having this all broken up lets you do that. And then there's there's like skill files that we're making and the skill files for VFE are in there. So that when you're running this and you're using an LLM like we all do now and it sees this and there's a problem with optimization, it can look through the skill file that comes with it and be like, Well, this happened. This is how you fix it. And it just knows how to do it immediately. And so because there's a few common modes, right? And so You can either, you know, you can start with dot fit, pg dot fit and just do it, or break it down and be able to hold different parameters fixed or not. And yeah, just being able to do that quickly and then also being able to hop from V F E to some other approximation and change your priors kind of quickly, just so that the workflow of actually using these things goes smoothly for you. Mm-hmm. That's kind of what we're going for.
speaker-0: Yeah yeah. Yeah, I I love that. And since you've opened the kind of worms of agents in AI, that's great. So I'm gonna I'm gonna play us out ⁓ with that. Well done, Bill. ⁓ but so first thanks all Jesse for this ⁓ Jupyter notebook and and you've proven once again that ⁓ you're such a boomer because ⁓ you're still using Jupyter notebooks. Like that it's not even Jupyter Lab.
speaker-1: Yes.
speaker-0: It's like Jupiter notebooks, like twenty tens vibe. ⁓ in the age of AI agents, that's ⁓ well well well done, ⁓ I appreciate
speaker-1: With your ⁓ C P server for Jupiter.
speaker-2: I did. I wrote my own I wrote my own Jupyter classic MPC ⁓ C P server. So if anybody else wants to boom around with me, Jesse Grobowski slash ⁓ NB NB ⁓ C P bridge, ⁓ it installs the server and it installs the bridge and you can collaborate with ⁓ with an agent from your cloud code terminal and everything lands right in your Jupyter classic session.
speaker-0: Hmm, nice. How how is your experience with the Jupyter notebooks and and AI agents?
speaker-2: So Bill is the one who turned me on, so that came out that came out weird. You wanna believe something? Believe that.
speaker-0: Yeah, I bet ⁓ I mean we're still talking about Jupyter notebooks.
speaker-2: Someone who turned me on to AI and Jupyter Notebooks and there was a blog post from I think the the author of NB Convert, like I got an email like a day later and he was saying there's a stack you can use and you can plug there's a there's an MCP bridge for doing like collaboration, like multiplayer notebooks, and then you're gonna plug in your agent and then the agent can work in the notebook. And I tried it out and it was just miserable and it was just terrible. I hate Jupyter Lab. I don't know how anybody can use it. Like you run a cell and it jumps all the way to the bottom, and then you gotta scroll back up to see what the output is. Like by default, it doesn't have like comp like closing parentheses automatically. You can't do the like question mark to get the like little pop-up at the bottom that shows you the health. Like maybe, maybe I'm absolutely like Stockholm syndrome, but it just seems like so much worse than like the old school like bare bones HTML thing. So I said, you know what, I've got a I've got an agent, I've got a Claude Max account. Give me ⁓ give me an MCP server for this. And it like one shot it and it was really good. And I've been sort of tinkering with it and improving it ever since. ⁓ I haven't had any problems with dropping. I haven't had any problems with like clobbering each other's stuff. There's other problems, right? Like it's janky. It's the same as everybody's hobbyist AI but it's like my janky software, so I'm willing to overlook the issues. I think if somebody else does it, they'll say like this is crap.
speaker-0: Yeah. It's the Ikea effect. It's like you you built it, so it's got much more value to your eyes than than to others. but so d I'm curious, do you do do you do a lot of plot ⁓ a lot of plots? Plots? I think you say plots, right? It is hard for me. ⁓ plots. Do you do a lot of plots in in Jupyter that you do with the agent that the agent is supposed to do? And how does that work? Yeah. What's your experience?
speaker-2: ⁓ it it does a good job. I ha I what I ended up doing was I strip all of the the images when I send the sort of blob to the agent. So by default it'll just like do its best and then give a thing. But then the it has a tool call where it can specifically say like give me back the PNG and then it'll look at it and it can make ⁓ improvements on it. So ⁓ I have softened like I got an obsidian vault so I'm not like completely stuck in two thousand and two. And you can just like rapid fire like plots into an obsidian vault with like some markdown and sort of like DIY notebook. ⁓ and that's really that's really great. the reason why I like great. I every plot I showed you just now was written by Claude. You just have to have you know you have to have you have to have taste. You tell it, you know, I'm French and therefore I need something aesthetic and if you show me something bad, I'm turning you off. And then it gives you what you want.
speaker-0: Because I might I mean so because to me, in my experience, even if I give it like a reference notebook, it will come back with plots which are absolutely a monstrosity and like but stupid stuff like the title will be phone size one hundred and forty eight. And then it will come back and be like, Yep, all done. ⁓ I followed your example, the notebook is ready for you to review. And I'm like No is d it's Absolutely terrible. So th how do you manage to ⁓ I if you manage to make that better, I need to hear about the solution.
speaker-2: Sure, sure. And I mean, part of the answer, like and part of the reason my notebooks were good then and they're good now is because the code's right there and the output's right there. Right. So you really I when I just want to like fire off a bunch of prototypes into a folder, like that's fine. But if I really care about something, I want to be close to the code. And a notebook means that the agent will put the plotting code right there. It'll make the thing, it'll have font size 170, and then instead of typing, I'd like a A dissertation to the thing about how how dare you have put the title so large and need you to do it again. Yeah, just do it. I just go into the code and I say make it eighteen.
speaker-1: Yeah. Yeah. Yeah.
speaker-0: Yeah. So that's where like stuff like if you do that in VS Code for instance, that's the stuff where you would do like the line edit or things like that. Yeah yeah. Okay. So but Tiki but that means you take back the steering wheel. Because that's what I end up doing. Basically I just use the agent for the first version. I know that for most of the code's gonna be fine, let's say eighty percent of the way there. So, you know, like for technicals that stuff. But then for plots If it gets to fifty percent of the way I'm happy. And and then I take the wheel back because just like basically it seems like at least for now, plots with agents is like trying to drive your Tesla on sinuous mountain roads. You know, it's like it starts to do weird stuff that can get you killed and so you're like, ⁓ wait, wait, wait, give me back the steering wheel. ⁓ it is definitely not, you know, like the the one on one where just you do full self driving mode, you're good.
speaker-1: No,
speaker-0: Yeah. Okay. So I thought I'm not completely stupid. I'm not having the worst experience with agents and and plots. That's reassuring for tonight. What about you, Bill? Yeah. Because you told me before the show that actually PTGP, one of the motivations to do that was ⁓ interactions with AI agents. So can you elaborate on that?
speaker-1: No, I Yeah, I mean I mean for for one, I wouldn't be w even doing this. I mean, this has been something that's been kicking around in my head for years, where I'm like, ⁓ if I if someone would just pay me a salary for like two years or longer to just do this thing for fun, that would be awesome. But that's not not gonna happen ever, you know? And so Now that this exists and it's kind of good enough, this is it's it's good enough to build these things. And and I, you know, I guess I do this for work, but I kind of like I kind of hate programming. I hate fixing bugs and I hate like getting stuck in little details that I just want the stupid thing to work because I want to think about whatever problem I'm trying to solve, you know? And it's great for that. So, you know, that it the fact that that stuff exists now is kind of why I could work on this project, but it also kind of got me thinking. It was like, well, if you can just have the robot generate whatever code you want, why do you even need to make a library? Why don't you just make it do the thing you want it to do? And then so I guess it was like, well, unless it can kind of just one shot different GPs and all these back end I mean, it's I think there's still a there's obviously still a place where libraries can you you still need it, of course, but But the fact is that most people are going to be act interacting with it, kind of with an agent in the middle, you know? and so I was thinking, how how should a new library be made in that context? And ⁓ I think I this is what I'm thinking now on and, you know, subject to change, because it's all new, but you know, the the documentation should be, you know, the thing can read all the code. You can just feed it all the code and it can read it and The doc strings are just there to to sort of talk about intent and what it's for. And humans probably aren't going to read it too much. And then but the documentation should be really nice to look at and good for people to read and conscious of people's attention spans because we have much we have much shorter context windows than LLNs do, and there's many other fun things to do that give us dopamine besides reading code all day, at least me. And yeah. Yeah, and it's and so like you know, that's what I was gonna ha so having these skill files, right? Instead of having extensive documentation, I'm like, here's a notebook showing this problem, here's a notebook showing this problem and how to fix it. It can be in skill files. And then when you run across it,
speaker-0: We have it, okay.
speaker-1: the thing, you know, you pointed to the skill files that come with that ship with PTGP and it knows what to do. ⁓ and another thing that is kind of interesting is, you know, when you're using it to write these GP approximations, I mean I'll all they've all been written in other places because I'm not putting anything new in here. So how to cite stuff kind of prominently so that other people's work is actually in there and I'm not just like, you know, vulturing around and cobbling together a bunch of other people's hard work that they had to do real math and think really hard about how to make it. So that's so that's kind of important, I think, for this. ⁓ but but yeah, so it's I think skill files are a really nice way to to sort of debug. And then and then I also think you can kind of interact with the I also think people aren't going to s at least me, I I w I don't sit and just like read documentation, right? Like top to bottom, beginning to end. But you can kind have the LM read it and then it's kind of like choose your own adventure where you can ask a question and then it gives you the explanation and the explanation comes from reading it and and ⁓ you know, you still look at it, but yeah, so I just yeah, how to make a a library that's going to be, you know, has a a purpose before before the LMs can just one shot anything that pops into your head, you know. Yeah. ⁓ whenever that happens, if ever. ⁓ And so it's just like stuff that's tested and it works and stuff that's trickier to just sort of one-shot and you know have have a few kind of like likelihood functions in there, but the stuff that's easy to extend, let the L like have PTG be kind of be ready for people to want to extend it ⁓ to and make it sort of open to that and sort of say where you could extend it and what it has and so I don't know. It's you can tell I'm still still thinking it through. But ⁓ but yeah I think it's interesting though.
speaker-0: I should hope so. It's not like the technology set in stone so
speaker-1: No. Right. Nobody knows, right? Yeah.
speaker-0: Yeah. ⁓ but that's so that's super interesting. You you have a skill file that ships with the package when you peep install it. And look at that. That's like it even Riverside is like super super happy with that.
speaker-1: Exciting.
speaker-0: So you have that in the skill file, so basically that's you going into the skill file and you wrote, Okay, like basically debugging steps where where it's yeah, well, you're gonna work on models that ⁓ that looks like that look like that. Often a problem gonna it's gonna be that. So you're gonna solve it like that way and then iteratively like that, and that's what the skill file is about.
speaker-1: Right. So there's one in there now. Let me see, where was it? But it's ⁓ it's ⁓ for ⁓ the VFE approximation and kind of all the issues that that you can hit just with that one approximation. ⁓ yeah, it's in docs agents PTGP VFE. There's a folder for that. And there's ⁓
speaker-0: Amazing.
speaker-1: There's a skill file for that. And then there's scripts for like making plots to see like preset plots to sort of help you debug stuff. So, you know, here's ⁓ here's a bunch of pitfalls with like links to like each section in the skill file, right? And it'll be like Sigma collapse, Sigma drops to zero, elbow appears to improve, it fits degenerate. Sigma grows during training while elbow plateaus and The condition number of your approximated whatever because of how you chose the induction inducing points is too high. And so there's a cutoff or you have really small eigenvalues there. Because this is all the stuff that happens when you're fitting GP models and you go and I think p I think part of the reason people don't use GPs is they hit one of these things and they go, ⁓ GPs suck. The test error is high. Screw this, I'm back to XGBoost or something, right? And But really there was just like a weird little thing that happened and it's something that you could fix. ⁓ so there's lots of lots of these sort of issues and ⁓ I you know I like to work on I like to work on problems while working on this so that I hit these things and can fix them and keep track of them. and so it's ⁓ yeah, it's like it's a place f where you can kind of collect these things because I think there's a lot of just like there's a lot of folk folk knowledge about how to fix Bayesian models, like when you see, ⁓ the, you know, my my look at my chains and they're in two different places like this. That means I have, you know, multimodal something's going on, but, you know, I should fix the prior because there's an identifiability issue. And ⁓ you know, I think there's a lot of that with GPs out there too, that's you know, maybe more since they're more exp obscure, they're hard it's harder to find that stuff. ⁓ so I want to have that kind of come with it. So that, you know, you don't have to be an expert in this stuff, but it can kind of guide you and so you so you do learn because I you know, I still think you can't run this stuff without knowing what you're doing, right? so it's like Yeah, that's the issue. Exactly. And then you do something really crazy. ⁓ but ⁓ but yeah, so it's it's kind of that in between that
speaker-0: You can run it, that's the problem.
speaker-1: Mm-hmm. Can kinda help people learn and and do it and kinda that balance between, you know, knowing enough to be dangerous and then also having the guardrails kinda built in and kinda help tells you why. So
speaker-0: Yeah. Yeah, I I think that's that's amazing and that's gonna be super helpful to people. So make sure to add all these links to the show notes. ⁓ Jesse, I do want to ask you about, you know, your like the good practices you've seen ⁓ arise from your work and and people's work with interactions with AI agents and how they fit into your work because I do I know you do so many things And you know so many things and so many literatures and so on, so I'm really curious to hear about, you know, what you recommend people to do and what you recommend not to do.
speaker-1: Jupiter notebook.
speaker-2: You should put it in notebooks, that's right. I guess I'll start I'll start with a liter literary reference since you've talked me up. You've heard of this like top like ⁓ Adam Smith had this notion of like a double coincidence, like double miracle. ⁓ it was in the context of barter economies where they say like a barter economy could never work because you need someone who needs exactly a sheep and who has exactly a goat, right? And then we can we can exchange those things, right? Or you're a haircutter and so you're willing to cut hair and you need exactly a pound of wheat, and I happen to be a wheat farmer, so we can do it.
speaker-0: Yeah yeah. So it's basically Catan. Like you you know the the the game.
speaker-2: Yeah, everyone's get to end the microphone. Real life. Exactly, exactly. And you think about think about like developers, like statistical open source developers. Right? Like there's not a double miracle. There's like a hex tuple, like octuple miracle. Because you have to be interested in statistics and like using models to solve a problem. But that's not enough. You have to be interested in the the methodology itself. Like you can't be someone who just wants to like pip install something and run it and get an answer and move on with your life. Like you're You're as interested in the tooling itself as you are as interested in solving some difficult problem. Then in addition to that, you need to be interested in programming per se, because now you're getting into like Git and workflows and collaborating with other people and writing code. And then in addition to that, you need to be a communicator because now you need to write documentation and you have to make sure that you have examples. And then in addition to that, you need to be a public speaker because now you're going to Pi data and you're talking about your stuff and you're promoting yourself and you're putting yourself out there. And like the the the number of people that sit in the middle of that Venn diagram has to be measure zero. And I think one of the powers of LLMs, one of the superpowers of LLMs is it lets you pick your spot and say, I'm someone who's really interested in algorithms. And I'm gonna really focus in on algorithms, and I don't care so much about writing examples. Like Claude, write me an example. First try, no mistakes. And you get a nice example, and you can really dial in on ⁓ focusing on like the algorithms that really motivate you to work. Or you're someone who's really, really interested in studying some specific applied literature, and you come to a Bayesian model and you say, I want to give this a try, and you run it and it doesn't fit, and you get an error message that says, ⁓ your model's not fitting, you should reparameterize it. And you think What are you talking about? Re-parameterize it. I have no earthly idea. And so now you can go to Claude and you can say, Claude, you it says I need to re-parameterize it. What do I do? And it tells you about the difference between a centered and a non-centered parameterization. Or it tells you about the pitfall of having a grand mean and local means and a random effects model. And like, ⁓ silly me, I put a I put a location parameter in all of my random effects. I need to just have one. Right? All of this folk wisdom, right, that kind of Bill is alluding to gets balled up into this big pile of weights. And it lets you say, I want to be the world's foremost expert on schools of fish. And I'm going to use a Bayesian, I'm going to use a GP to estimate the locations of schools of fish at a certain time of year in the Pacific Ocean. And I'm going to be able to do a really, really good job on that because I have access to this side brain that can sort of take on the job of knowing the ins and outs of that piece. My perspective on LLMs fundamentally is that everything is an arbitrage between learning something and doing something now. And if you go like full Gary Tan and you just LLM nonstop, everything is an optimization problem like G stack, let's go, then you've sacrificed every opportunity you have to learn something new. ⁓ on the other hand, if you put on the purity ring and you say like LLM bad, AI is ⁓ taking all the water. then you've completely left on the table the opportunity to build incredible things and to move at pay at really fast speeds in those zones where you've decided for yourself I don't want to be a domain expert in that. But I still want to be able to make progress in my problem. And modern problems require that you have this like octuple miracle of skills and interests in order to make progress on them. So ⁓ I'm I'm super bullish on technology. I love it. I also have felt myself getting dumber. Like I've felt myself making the wrong choices on that split between learning something and doing something because your boss comes in and says, I need this yesterday, like fire it off. No mistakes, first try. And I'm like, well, I guess I'm putting it out with Claude because I do what I have to do. And and I miss those days where I would sit down with a paper and I would get a pencil out and I would rederive the whole thing and really feel like I understood it deeply. And now it's just like, you know, Claude, read this, give me a summary, like let's put it into Pi MC. So there's a trade off there. You have to be cognizant of the trade off, but the same time, like your reach just grows enormously, tremendously, and it's incredible.
speaker-0: Yeah. Yeah. And I feel like so you know what, you should you should come back on the show to talk about that, you know. Like how you use AI agents and and how how that goes into your workflow and the kind of models and so on. And the gotchas. I think the gotchas are the most important because at least in my experience from what I see, I feel like now more and more one of the main skills pun intended. Is is to ⁓ To be able to filter the the LLM's output. And I feel like, you know, LLMs are a bit like YouTube ⁓ the YouTube algorithm where if you wanna learn something, it can sometimes give you some absolutely incredible gems and it's gonna work and that's gonna be awesome. But a lot of the time it's also gonna give you the bullshit, conspiracy theories, videos that look like they are credible and they are not at all. And you actually need to be able to make the to make the difference and that's why to tighten your point, Bill, being an expert in what you're building is much, much easier to do that.
speaker-1: Yeah, no, it's true. I mean, it kinda comes down to some simple stuff at some point too, is they they will just do what you ask them to do, right? And you have to think really carefully about what you're asking it to do and and this thing you're picturing in your head and then what you actually wrote, there's usually a pretty big difference between those things, you know. Yeah. If you were to pass it to someone else and and then you know, of course it's not always gonna do it faithfully either, but ⁓ yeah. It's it's ⁓ it's an interesting problem. It's it's not unlike working with someone else, except you you sort of have to get to know them and what they're good at and what they're not good at and how to work with them effectively and and they're usually, you know, it just like working with people, they can be w far better than you at this thing or that thing, and yeah. Maybe you're a little better at this other thing for a little while and ⁓ and just it's like a how to collaborate with this weird little thing that works very quickly and can make a lot of mistakes quickly and make a big pile of crap very quickly. Or you can do something interesting, right? And you kinda have to not be lazy and say, yes, yes, yes, yes, yes, yes, yes. And which is hard to do sometimes, you know.
speaker-0: No, for sure. For sure. For sure. Especially under time pressure. And like this, I think it's even harder than be thinking a lot about how you ask what you want. Because then you can ask it exactly the way you want it, but then at least two things can happen. It can come back and tell you if d it's done it, but you know, it just thinks it's done it and then it didn't. It literally happened to me several times. and also it can happen that It cannot somewhere in the thinking chain, for some reason cannot answer the question you ask it. But then since it wants to come back with an answer, it will switch what it's gonna answer, but you know, like ever so slightly. And it's gonna be a related question. It can be a good one, but it's gonna be a related question to what you ask, and it's so close to what you asked that it's gonna take you a while to understand that it's not exactly what you asked and and that's like this is very tricky. ⁓ but I think Jesse you wanted to to add something for a while, so I'll I'll give you the floor.
speaker-2: I just wanted to you mentioned the YouTube algorithm and how YouTube has on one hand these incredible learning resources and on the other hand it has just slop. I think there's a more pernicious failure mode that is relevant to LLMs, which is you take something like ThreeBlue One Brown, who's one of my favorite channels on YouTube, does math videos, amazing stuff, and you sit on your sofa in your pajamas with a cup of ice cream. And you watch a video about how to solve an integral with animations, you think, wow, that's incredible. I've learned so much. And then you think, I'm so smart, I'm so great. I watched a video about integrals, and then you close it and you play, you know, whatever. And you have this illusion that you've learned something and that you're like a sophistiqué and everything is delightful, when in reality, you're still a knuckle-dragging crow magnate. And LLMs have the same failure mode where you can convince yourself, like, I made this. You know, like I you shoot off a dashboard and it looks really nice, and you're like, I made a dashboard. And you have to realize that, like, no, no, no, no, no, no, no, no, no, no. You prompted an LLM, you got an interesting response out of it, but you haven't learned anything, you haven't built anything, you got something useful. Right. And you can't you can't lie to yourself and act as though you've improved yourself. by having done that. I think that's that's quite dangerous and it's something that you have to stay on alert for.
speaker-1: Yeah. It's something that's happened and it's like a a thing in data science that's been around a long time is you can write a model, get a fit, boom, done, don't think about it, but then well actually the this feature that comes in, there's something wrong with it. And so your model always predicts this thing when that feature takes a certain value because you didn't look closely enough at things. Right. And you didn't actually learn about this thing that you're modeling, right? Mm-hmm. And I think I think it's nice to think about models as not a tool for doing anything, but like a mechanism for learning about the thing that you're feeding into it, you know. And it it especially with LM's able to just pump stuff out, like you still need to be super critical and care about getting things right. Right. And that's that's kind of difficult to do, you know. 'Cause you can yes, just sit there and feel like, ⁓ look at all the stuff I did and look at all I learned and The thing is the code is running and it doesn't error. It's done, you know? And you know, sometimes you can get great results and and that's the sign that there's a problem you know. And all all this sort of stuff. And I think I don't know if if this it's like this problem's always been around, but L Ms just kind of pour gasoline on it, you know.
speaker-0: Yeah. Yeah, no, exactly. I think it's like d and that's a very good point. I had never thought about that, but yeah, it's a very related problem to what we had before. But now it's like yeah, it gets even worse because it's just like bigger dimension. So Yeah, yeah, yeah. That's yeah.
speaker-1: But they'd be great. I mean, I was working on a icause you sometimes you don't need to know a lot about something too, or and and you need to sort of make a note to maybe I don't know, ask someone who knows or whatever. But like, you know, I was working on an example for PTGP and it was on like ⁓ you know, water main brakes and there's water mains are made out of different pipe materials. And I know absolutely nothing about that, and I don't know anybody who knows anything about that. But LMs know about water main pipe materials, right? And in a pretty straightforward way, like, what is this type of plastic? You know, like I trust the answer that will come out of it for something like that, right? That's pretty like I imagine that's pretty basic for people that know about water mains and how to do water stuff and people that know pipes, right? So it's like it's like basic stuff from other fields. Now you just have access to that, which is really, really nice. And That kind of thing really can help you move on because ⁓ yeah, it's it's it's like I've seen people talk like, Well, models that are really specific and just just trained on this one field are are useful. But also the fact that models are trained on everything is also really, really useful too, because there's there's all these connections between different things and that can be in there. So
speaker-0: Yeah. No, that's true. Yeah, I just think that basically it's like it it lowers the it lowers the the the floor to to entry, the barrier to entry, and like and it also increases the ceiling of what you can achieve, but that also means the standard deviation can get way, way bigger on on what you can get. So ⁓ Yes. ⁓ so I guess we'll talk about that again in the show, like almost every episode now. ⁓ it's in our life. ⁓
speaker-1: Yes.
speaker-0: But I need to to to get you to let you guys go, especially Jesse, it's very late for you, so ⁓ thank you so much for for being on the show guys. I think it was awesome. And Bill, I have to ask you the last questions, of course. So first one, if you had unlimited time and resources, which problem would you try to solve?
speaker-1: ⁓ so I don't know ⁓ what the other answers are, but I'll I'll go with ⁓ I'll take you very literally at unlimited time and unlimited resources and say ⁓ would be really cool to to figure out what the heck's going on in outer space. Because ⁓ hey, we just get to look at it with telescopes and do science like that.
speaker-0: Uh-huh. Yeah.
speaker-1: And it's just little bits of tiny, tiny little bits of photons that come off of a very big place that we we we can't know anything about that. And ⁓ if you had unlimited time and unlimited resources, you could probably learn something about what's going on. So I think that'd be cool. And and world peace and and world hunger.
speaker-0: No, no, no. You just get one and Second question, if you could have dinner with any great scientific mind, dead, alive or fictional, who would it be?
speaker-1: Well, this one's probably a popular answer, but I'm gonna say Richard Feynman. one of the one of the things that got me into science or whatever and switched my major in college when I was a aimless youth, I read ⁓ Feynman's Rainbow, which is a really nice book about that his one of his I think his first year or it he some he was a grad student of his worked with him. ⁓ wrote. I haven't read it in a long time, so I probably got that it's something like that. But it's a really nice book and really inspiring because he's sort of a a person that ⁓ put a lot of passion into a lot of different areas and enjoyed learning about stuff. And ⁓ yeah, so he's sort of a inspirational figure in that way and had a lot of kind of varied interests and ⁓ you know, w life is short. We all have a limited amount of time here and ⁓ you know, you gotta make the best use of it. He's like a Kind of a good, interesting example of of that and kind of appreciating the things that you get to learn about while you're here. So yeah, I don't know, it'd be cool to have dinner with them.
speaker-0: None of that. Yeah. Topical ⁓ answers. Like basically you stayed on the same topic for a
speaker-1: I guess so. Well done. Well done.
speaker-0: Yeah. Impressive. Awesome. ⁓ well guys, I think ⁓ it's time to to call it a show, but ⁓ yeah, ⁓ really awesome to have you here. Jesse, I guess you'll you'll come back on the show very soon. ⁓ we'll talk about all these all this stuff. Be also come back, you know, whenever you have ⁓ you have some new stuff to share. You're always welcome.
speaker-1: Sure. Yeah. I'll start begging, you know, maybe tomorrow again.
speaker-0: Yeah, baking again. Yeah.
speaker-2: This is this is this is Laplace T V, you know, so I'm always happy to
speaker-1: I'm always a reach. ⁓
speaker-0: Where would you go? I mean apparently Bill has hobbies but ⁓ Jesse and I don't, so you know, we're always available.
speaker-1: Mm-hmm.
speaker-0: Awesome. Well guys to to have you on the show. ⁓ listeners take a look at the show notes, they're gonna be great for this one also. And ⁓ well, Bill, JC, thanks again for taking the time and being on the show.
speaker-1: Thank you. Thanks so much.
speaker-0: This has been another episode of Learning Bayesian Statistics. Be sure to rate, review, and follow the show on your favorite podcatcher and visit learnbasetats.com for more resources about today's topics, as well as access to more episodes to help you reach true Bayesian state of mind. That's LearnBaseTats.com. Our theme music is GoodBayan by Beba Brinkman. Fit MC NAS and Mega Rand. Check out his awesome work at Bebabrinkman.com. I'm your host. Alexandora. You can follow me on Twitter at Alex underscore Andora like the country. You can support the show and unlock exclusive benefits by visiting patreon.com slash learnbase tax. Thank you so much for listening and for your support. You're truly a good baby and change your predictions after taking information in. And if you're thinking I'll be less than a maze. Let's adjust those expectations. Let me show you how to be a good daisy. Change calculations after taking fresh data. Those predictions that your brain is making. Let's get them on a solid foundation.
speaker-1: Huh?
speaker-2: Yeah. I improved my background. There were complaints about last time I had like a dirty sock stuck in Shanghai. My internet connection was terrible. ⁓ so here
speaker-0: Yeah yeah that's true.
speaker-2: Yeah. It was like some behind the scenes that my connection was so terrible, I think it took us three hours to record that and then your editor guy was just like, What are you doing?
speaker-0: Yeah, yeah, I remember now. Yeah, yeah, yeah. I mean, but state space models are worth it, so you know. ⁓ so that's fine. You know. ⁓ something. Yeah, yeah, exactly. but thanks again for joining. Live from Chicago. It's ⁓ starting to get late for you, so ⁓ thanks a lot. We we appreciate it with ⁓ with Bill. ⁓ Bill, you're live from Portland, Oregon. I remember the background. ⁓ from the time we we used to work together.
speaker-2: But thin's the juice is worth a squeeze.
speaker-0: ⁓ so ⁓ let's ⁓ let's talk about you actually because ⁓ it's your first time on the show, but finally it's ⁓ it's a long overdue episode. I managed to lure you in, of course, with both GP and Jesse. So
speaker-1: I begged and begged and begged and you finally let me come on. So thank you. Thank you. So yeah.
speaker-0: Bill is making me look good here because I was the one begging. So ⁓ but thanks ⁓ Bill for ⁓ for saving my feelings. yeah, actually can you ⁓ let's do your ⁓ your origin story, you know? ⁓ how did you come to what you're doing today? You know, you now you do a lot of base stats and you do a lot of Gaussian processes. ⁓ you're like one of the Yeah, you're like the GP guy from the Pime C team, but How did that happen? Like where where did you go wrong?
speaker-1: Where did I go wrong? Man. Yeah. So that's actually a great leadoff, because ⁓ I ran into this stuff for the first time a long, long time ago when I was an undergrad at the University of Oregon. And it wasn't in a stats class or anything, but I was I was working on a physics research project and it was with the LIGO experiment. and the task was detecting Gravitational waves from core collapse supernova. And they use Bayesian methods there. So that's how I got introduced to what Bayesian stuff was. I remember at that time I thought statistics was just like you have dots and then you draw a line through them. Mm-hmm. And it was kind of really boring. You just why don't you just draw the line? So that was very eye-opening. Yeah, so that was a A Bayesian inference problem kind of at the core. That was my first introduction to Bayesian stuff and then working on that is also what led me to Gaussian processes actually kind of early before a lot of other stats stuff, which is which is interesting, I guess. A weird way to find them.
speaker-0: Yeah. Yeah, so you started working first on on Gaussian processes?
speaker-1: Kind of early on I was just I I just really liked working on this project and got kind of stuck on it and it was more fun than physics homework. And ⁓ you know, eventually the problem was was really that you had to make a a surrogate model for these simulations of core collapse supernova and GPs are used for surrogate models, but I you know, it was like learning about, ⁓ what's maybe machine learning, what kind of statistics? It's gotta be some kind of regression thing and yeah, and then kinda just stumbled across Gaussian processes like that. And then ⁓ yeah, that's that's how I was introduced to them. So before I learned much about statistics or machine learning.
speaker-0: Yeah, okay. Well, okay, so how is it? But is it's not the easiest no entry door to the world of snacks.
speaker-1: It was very difficult. But yeah, I spent a lot of time. It was really interesting. I spent a lot of time learning about it and I just eventually thought this was a lot more interesting, well, than the physics stuff, physics homework at the time, you know, and and kind of decided to go that direction instead of physics. And so ⁓ took a few stats classes, took a machine learning class and went ⁓ graduated and then went to ⁓ Went to get a job and doing data stuff after that. Yeah. But I don't think I had it all figured out then. I just barely kinda understood GP's.
speaker-0: Wow, damn. Right. Yeah, yeah.
speaker-1: But it was interesting to learn about other stats after seeing those because, you know, everybody talks about the special case of G P is regression, a special case of GPs are neural networks, and a special case of, you know, time series, AR models, whatever. And so they they have connections all over the place. So it was ⁓ it's kind of fun to learn statistics from working backwards like that. Yeah. And also learning more about GPs at the same time. So they've always sort of been the little touchstone for For something I can come back to or a a way for me to understand something, thinking about kernels.
speaker-0: Yeah. Yeah, I mean and thankfully you did that because then I learned a lot of GP stuff thanks to you. ⁓ basically thanks to your production on the on the Binsey side, you know. Tutorials, you know, ⁓ of like all the things I I read all of that, ⁓ the code for the the sub package, because at the time it went you still had to read and write code. ⁓ yeah. Yeah. Incredible. Yeah, it was another world. and ⁓ and yeah, like I mean I learned so much from all the Pine C G P sub package that you wrote. So thanks Lord for doing that. ⁓ and I
speaker-1: Back in the dark ages. Thank you, thank you. I'm glad ⁓ glad people seem to like it. So Yeah. It was really fun to work on. So I appreciate that.
speaker-0: Yeah, I'm sure I'm sure a lot of people will be happy to hear about you today because I've had a lot of ⁓ of people asking me to have ⁓ to have you on the show to talk about GPs, so ⁓ you know, I'm sure a lot of people will be very happy ⁓ in the in the community. How did actually so you wrote the sub package for GP four point C
speaker-1: Cool, I'll do my best.
speaker-0: During ⁓ Google Summer code, is that correct?
speaker-1: That's right. Yeah. Yeah. ⁓ so yeah, Google Summer of Code was great, by the way. I ⁓ it was really nice to have a period of time to get paid a little bit over the summer. I this was during my I was doing a masters in statistics when I when I did that. And it was also really nice to ⁓ work on some code, like a code project, which I really missed while doing my masters because it's all like math on paper. And I missed coding things that, you know, did something and were used by people. So it was ⁓ it was a great time to do that. But ⁓ but yeah, the the reason I got interested in Py ⁓ C before that was that same research project because I needed a software that you could do it's really specific. You could auto-diff through a Fourier transform, you could set priors and do stuff like that, and it was really easy to kind of extend and PyMC fits the bill there and and Theano, right? They know was the back end of PyMC at the time and it had Fourier transforms in it. And I don't think other stuff, at least that I could find at the time, did. So and doing gradients through a Fourier transform and then a GP was ⁓ ab above my skill level, let's just say. So Yeah, it was like the perfect fit to to look at that. So that's kind of how I got interested in Py ⁓ C and it was easy to experiment a lot and then was doing summer of code, it was like, well maybe some of the stuff I could ⁓ you know, include it in Py ⁓ C and so it looked like I got a lot of work done really quickly, but a lot of it was kind of floating around on my computer already and it was just ⁓ a lot of cleaning it up and making it nice and making a good API for it. ⁓ but yeah, so that's that's how I kind of Submitted that to Pi ⁓ C.
speaker-0: I mean, you had to write the code anyways before, so you know, it's still ⁓ yeah, it's still your
speaker-1: Round, yeah. But I mean the having the mentors and working with like Chris Fonisbeck and the other folks that helped me out at the time was super, super helpful to 'cause that was the first time I'd, you know, written code that had to be like good, that was gonna be used by more than a couple of people and was like visible in public. And so, yeah, it was great to learn about how how to do that.
speaker-0: Yeah, yeah, no, I mean I'm I'm guessing it was nice and and yeah, Chris is is such a great mentor and teacher that it's just yeah, my each time I get to work with him I'm I'm super happy and excited, honestly. Yeah.
speaker-1: Yeah, just learned a ton, super encouraging and yeah, it was great.
speaker-0: Yeah, honestly. and so that's that's how you you started the the G P route on Prime C. ⁓ and then like did you do you have any let's say, you know, stories from the trenches in how you used GPs for clients for models in in real life, ⁓ and and maybe advice you can give to to listeners who are curious about them but haven't tried them out yet.
speaker-1: Yeah. ⁓ yeah, I've snuck them into a few models over the years. ⁓ working at so I worked at a kind of a series of I guess just straightforward data science jobs. They weren't, you know, doing Bayesian stuff in particular or or whatever. And yeah, there were always opportunities to use to use Bayesian statistics. ⁓ and then and then GPs can fit into that. Because one of the reasons I liked PyMC was that You know, there's kind of the split. There's sort of like the GP model as like a machine learning algorithm. And you can also put sort of embed GPs inside of a you know, like a hierarchical model, like something you would make in Pi ⁓ C. And ⁓ and that sort of second thing I thought was a really nice thing to do that was kind of underused because there's it's really nice for that time when you have, you know, just some function. And you need to sort of fit it and it's within this larger model. And maybe you don't really care about what the result of it is. Maybe you do, maybe you don't. And you need to sort of get rid of that variance or separate that variance from whatever other thing you're modeling. And GPs are great for that because otherwise you maybe you're using polynomials or you're using splines or something like that. And you know, you can't predict outside of the window that those are based over outside the training data and then you have to futz around with the order of it and how to set the priors and it's just a bit of a rabbit hole. And GPs make that really slick where you'll just have a scale on it and then a length scale parameter and two knobs to turn and set priors on and all that model selection stuff just kind of falls out automatically. And so that's sort of why I liked, you know, using GPs in PyMC and ⁓ It was also nice to have that functionality built in so that, you know, what happens a lot when you're, you know, working on a Bayesian model is that you have to sort of build it up stepwise, right? Like the Bayesian workflow ideas and you sort of need to quickly be able to try stuff out and like, well, maybe this thing's a bit of a function. I could just throw a GP at it and see how it does. And just making that cycle faster for dealing with a little nonlinear function in your model is just a really handy little thing to have in your toolkit and something that's quick to use. So you know, there were there's definitely been times at at work where you're just working on kind of a regression model and you wanna use Bayesian stuff because you don't want to have to explain p values to anybody and the model itself is something not quite off the shelf and so you can just whip up a little Bayesian model and sample it and, you know, read the results off in a pretty straightforward way and ⁓ Yeah, help people out.
speaker-0: Yeah. Yeah, and I mean I resonate a lot with that for a practical view of GPs. ⁓ I think one of the best ways I understood them is yeah, like if you have some kind of function ⁓ that you don't really know what the form, the mathematical form is. ⁓ that's a good use case for a GP. ⁓ yeah. And that's a very good, usually very good first approximation. And then sure if you care about the actual formula, then yeah, ⁓ for sure. It's gonna it's probably gonna be better. ⁓ if not if not from a prediction perspective, at least from ⁓ computation perspective. huh. ⁓ GPs tend to be ⁓ tend to be computationally expensive, at least the vanilla ones. but yeah. Like basically, okay, I have some form of nonlinear ⁓ relationship here. let's try a GP, you know. and that's usually a super super helpful ⁓ super helpful approximation of what's really going on.
speaker-1: Yeah, because that's kind of one way to build them up is they're a prior over unknown functions. And you pick a kernel and the kernel is kind of like, what what variety of unknown functions are we talking about here? And yeah, and ⁓ you go from there, right? And ⁓ there's a few ways to build GPs up GPs up from you know, things that make sense. I like that version, and then I also another one that kind of comes in handy is that a GP is The same thing as a hierarchical model where maybe you have some continuous covariate, right? Like ⁓ one of the examples that people have in PyMC a lot is the it's like batting averages. It's one of those top line PyMC examples. And you know, a bunch of baseball players. Yeah, exactly. There you go. And say say you want to consider age as a covariate. So all the players now aren't exchangeable. They all have an age. So, okay, what do you do with that? Well, that's just a hierarchical model, except instead of all the players just being kind of independent players, they now have this age relationship. And that's you can model that with a GP with a length scale. And so it's just a slight step up from a hierarchical model, too. So there's a lot of ways you can sort of find yourself wanting to use a GP.
speaker-0: Yeah. Yeah, and you can do you can do hierarchical GPs. ⁓ so that's that's definitely super fun. ⁓ there are some tutorials I've actually we've been working on that together ⁓ with BL where there is a two part HSGP tutorial where it culminates in ⁓ in an example where we show you how to ⁓ build a hierarchical HSGP in PyMC. So that's exactly these kind of ⁓ these kind of of cases where you would have for instance ⁓ hierarchical higher level GP on age and and then each baseball player in your example would be ⁓ would be their own GP but that's ren ⁓ I mean that's sampled from The population of G Ps that we have on the population of players, which is the first G P the higher level one. So
speaker-1: Yeah, yeah, right. You can can go GPs GP's all the way down. Yeah.
speaker-0: Yeah. This is super cool. And and now is HSGP, so Hilbert Space Gaution Processes. I'll link also to a few episodes I have on that and the webinars. I think when I think yeah, Juan Arduz came on the show to explain the math really behind that. ⁓ but basically in a nutshell you wanna talk a bit about HSGP build because ⁓ I feel like it's been really a game changer for practitioners, honestly.
speaker-1: Yeah, yeah. No, this was ⁓ it was from a paper Solon and Sarka and then Yeah. Ru RuTort male. I'll probably just butcher the names horribly. Yeah. And that that second paper though was like basically like, ⁓ here's this thing from this other paper, and it would be really handy to embed in a probabilistic programming language because you it's it's just such a good fit because
speaker-0: Yes, yeah, yeah, yeah.
speaker-1: So there's lots of GP approximations out there. And this particular one works it works well for cases that happen a lot when you're working in something like PyMC and you want to throw a GP at some unknown function. So it works really well for low-dimensional things that have like a low-dimensional input, 1D, 2D, maybe 3D, but don't go higher than that. And when you're kind of using the common stationary kernels like exponentiated quadratic and the Matern ones. And those are yeah, super common. And you can kind of build these like, you know, generalized additive model type models with them. And they work really well in there because the approximation is it turns it into a linear model. So it looks a lot like a polynomial model or a spline model where you have this set of basis functions And then the weights for each basis function are parameterized by the length scale parameter. And you end up writing something that looks like a linear model and it samples really nicely. ⁓ it's ⁓ it's an approximation that works well with MCMC. And so having that sort of ready to go and easy to use in in PyMC was super, super handy. Because yeah, GPs are slow. That is their big problem. Yeah.
speaker-0: Yeah yeah yeah for sure. ⁓ so I'll link to a bunch of these tutorials and episodes in the show notes folks. ⁓ feel free to look into that. I know we're ⁓ throwing a lot of technical terms here. ⁓ kernel, length scale, amplitude. We haven't defined any of them.
speaker-1: Yes, sorry.
speaker-0: I mean it that's fine. ⁓ we'll we'll probably do it. ⁓ we'll see f w how how much in the details will go. ⁓ depends on what you wanna talk about and and show for a PTGP. But first before we do that, because we're ⁓ twenty minutes into the recording and I haven't had asked any question to Jesse, which is ⁓ very rude from me. ⁓ even even for a French guy that's rude outside. ⁓ so ⁓ yeah, Jesse So you've been on the show episode one twenty four, it's in the show notes, folks, in the related episodes, I really recommend it because if you wanna hear about Jesse's very interesting background, ⁓ very diverse and ⁓ very interesting that's gonna be on that one. Today it's gonna be all technical from ⁓ Mr. Grobowski. But yeah, like ⁓ what's your experience with GPs, Jesse actually? ⁓ they like do you have worked a lot on on them or are you here just to learn?
speaker-2: I'm I'm the ⁓ I'm here for moral support. Actually there's so Bill started with this funny quip that everything's a special case of a GP. There's a nice coincidence that a common filter is a sequential GP solver. So in a special case where your covariance matrix is sort of blocked diagonal because you have these temporal
speaker-1: He's Yeah.
speaker-2: relationships, instead of trying to invert the thing all in one shot, like you're forced to do in a most GP land, you say, well, I'll just take it one block at a time. I'll do my one little bit of solution, and then I'll sort of march down the diagonal solving the thing. But the the complete solution is what we in time series land call a calm and smoother that sort of takes this two-sided look at the time series and gives you back a smooth ⁓ function based on your model. So direct GP, I'm maybe much less experienced, maybe, much less experienced than Bill. ⁓ but there's this very nice way in which the things that I work on and the things that Bill work on tie off. The other nice coincidence is that a lot of the math well, not a coincidence, as a result of the fact that these are doing the same fundamental linear algebra operations, a lot of the optimizations and w work I do in the background on PyTensor ended up really benefiting Bill's work as well as the stuff that I was doing in state space. So if we have time, maybe we'll talk a little bit about some of the ⁓ linear algebra assumptions, some of the new number back end things, things you can do in PyTensor with number now that you can't even do in number with number, which is really exciting.
speaker-1: Yeah.
speaker-0: Yeah, I think it's a good time now to to dive a bit more into PTGP so because that's why I invited you guys on the show. You have this new package out. PT is for PyTensor, or at least I think. And GP I'm pretty sure is for Gush and Processes. Yeah. Yeah. ⁓ damn. That's some good inference on my end. Yeah.
speaker-1: Yes. Creative name. Creative name.
speaker-0: Yeah yeah. but so yeah, well what's up about that? Like what's the idea, what's the ⁓ elevator pitch, why is that even helpful and interesting to people, Jesse? And I think it's linked to what you were exactly talking about.
speaker-1: Yeah. So GP's on PyTensor. ⁓
speaker-0: Wait, that's Bill. So for listeners.
speaker-2: So I mean it's good it's good for Bill to answer this because PTGP's his baby. Let me just say that if you go out into the world and you say, I want to fit a GP, as you know, coming as a more of a novice GP person, it is not at all clear what you're supposed to do. I mean, there's PyMC and I love PyMC, so you can do PyMC, but PyMC is really locked into this full exact GP Framework until the HSGP stuff came along. But if you want to do four or five six D inputs, HSGP is off the table. Exact inference becomes off the table very quickly. So there's GPjax, there's GPflow, there's GPyTorch, and they all kind of have like little idiosyncrasies. And I think what Bill accomplished with PTGP and what I've been hashtag blessed to be a part of is to build something that from first principles says like, We're going to be somewhat opinionated and we're going to be useful. That this is like the practitioner's package.
speaker-0: Yeah. Yeah, I love that. ⁓ I feel like it defines you also quite well, Jesse. Opinionated and useful. ⁓ I think it's fine. ⁓
speaker-2: There's that is the nicest thing anyone has ever said to me.
speaker-0: That cannot be true. Bill, so yeah, like ⁓ the ⁓ tell us ⁓ tell us more about PTGP in well, I I interrupted you so
speaker-1: No, I'm glad because that was a much better thing than I was gonna ramble about, so now I'll start rambling. But yeah, I think it's true. GPs are hard to use and this is my I guess I'll say my kind of semi controversial thing is that like if if if GPs were really, really fast, they would be everywhere. People would use them all the time, and a lot of other models would just be kind of useless. You know? Yeah. Which is a funny thing to say in the age of LLMs and deep learning especially. But ⁓ just because there are every other model is special cases of them and they're interpretable and they kind of sit right in this Venn diagram between statistics and machine learning. Like if you were gonna color color the middle in, it'd be GPs. And ⁓ you know, we talked about having GPs and Py ⁓ C where you sort of embed them within a larger probabilistic model. There's also
speaker-0: Mm-hmm.
speaker-1: probably more often the case where you just sort of have GP on its own, the GP model. Like I have some data set and I need to make a predictive model for it. And I really care about uncertainty in the predictions and I want to have some level of interpretability of like why this model is doing what it's doing. ⁓ then GPs are a great answer in that situation too. And that's a it's a pretty common thing, especially when you need a model that's accurate. And uncertainty matters a lot.
speaker-0: Yeah. Yeah. I completely agree with that. I think ⁓ I think there is like a bad reason for not using GPs in the sense that it's the same that it's a limitation of the universe is ⁓ computational limitations. And I think a b a good reason to not use GPs is if you have the actual mathematical formula for your phenomenon and you can derive it and you actually want it and that actually helps solve your problem better and faster. But it's like a very small use case, I would say. ⁓
speaker-1: I wanna.
speaker-2: Can I can I jump in and say I feel like we've been a little mean to GPs up to now to say that they're merely functional approximators or like they're black box in the way that a spline model is. Like a spline model, that's my black box example. I'm such a boomer. Like ⁓ neural network or something.
speaker-0: Yeah, have you just one further to you know like easy to grab.
speaker-1: Shapes. Yeah.
speaker-2: Have you ever seen that meme from ⁓ it's called like ⁓ I forget the name of the c it's a it's a superhero comic and the the guy's talking to his kid and there's like a fighter jet flying in the background and he's look at what they have to do to imitate even a fraction of our power. And I feel like this is GPs and it's like splines. People do splines, like you get basis functions, or people do like ⁓ these has support vector machines with like radio basis functions or people do nearest neighbors. And it's all boiling down to like, I have this notion that things that are close in the input space are close in the output space. And that's like fundamentally, this is what GPs give you. And it gives you this really rich toolbox to define a notion of closeness and to say that in this case, close means this, and in that case, close means that. And you can it's not merely right, like putting down, lying down basis functions and then saying, like, well, whatever the data says is what the data says. And I think maybe this is what Bill means when he says they live at this very beautiful intersection point between machine learning, which is like let the data speak, and traditional statistics, which is let's sit down and think really carefully about a problem and write down a structural model of how to solve it. And plus they have this Bayesian like je ne sais quoi, which really, you know, just pushes it over the top.
speaker-0: Mm-hmm. Yeah. Yeah, yeah, yeah. ⁓ completely agree. And actually they are a fun black box in a way. I don't think so. I think describing this stem describing them as black boxes is not very Yeah, it's not accurate as you were saying, because they are just I would say it's like ⁓ let's ⁓ maybe there is a good analogy. It's like they they are like ⁓ Shredding Shreddinger's models. You know, they are black box until you open the box. and then and then it's like perfectly transparent because once you once you observe data, they just become multivite normals, so it's like a perfectly known object. And they also are extremely interpretable, which is kind of like the opposite of being a black box, ⁓ because the kernel function is exactly what you just explained, Jesse, where it's like, okay, what's my notion of what does closeness means for my problem? That's gonna give you the answer to that question is gonna give you hints ⁓ as to which kernel functions to look at and select. And then you had the amplitude and you had the length scale. So these ones also are meaningly interpretable. I think, Bill, ⁓ you can you can ⁓ give that interpretation for us here.
speaker-1: Yeah, yeah, right. Sure. I mean there's your there's your definition of a kernel, right? Is you want to go, okay, I got this data point and take my X values, and I got this other data point and take my X values, and then the kernel function says it's a function of those two data points, and it tells you, okay, well, how similar should the Y's be here? Which is a really interpretable thing. And I think it's something that a lot of people kind of naturally when they think about modeling. That's kinda how they think about it. It's in a lot of ways it's easier to think about closeness in the input space and how that relates to the output than it is to think about like, well, if I increase this variable, does that increase or decrease the y variable and by how much? You know? Because then you have to s separate the effect of everything. And sometimes that's really nice and you want to separate the effect of everything. And sometimes there's a lot of interactions that you want to sort of have in the soup with you there. And GPs are when you know, kernels are really great for sort of thinking about like that. The it's almost like a spectrum. It's like you have these additive models with the terms and that separates everything out. And that's the easiest additive model to make. And then you can add interaction terms as you make your regression model better and better. And then GPs kind of work from the other direction where You have this kernel over all the inputs and everything's interacting. But then if you want to make your model more complicated, you split it out and you start looking for individual effects. And so both of these things kind of work at a spectrum and then meet at a middle. And then, you know, GPs, you can have the additive terms in the mean function. They call it the mean function when they're working that way. But it's really just, you know, all these things live on a spectrum of you have an additive regression model with. errors that aren't correlated and then maybe some of the errors start getting correlated with interaction terms and then a GPU over on the other side where it's like everything's correlated and and you have everything interacting and you can kind of you know it's it's nice to be able to work anywhere within that and start from one end or the other. So and kernels let you do that. And so that's kind of ⁓ what a what a kernel function is. And they're all composable. You can add them together. You can multiply them ⁓ there's like a little kernel algebra. They have to kind of follow a set of basic rules. and they're just functions that dis define, like I said, just to restate how similar given if these two X points for two different data points, you know, take these two, how similar are they? Well then how similar should the the Y's be? And that's kind of what the kernel's telling you very roughly.
speaker-0: Mm-hmm. Yeah. And length scale? So length scale is a terrible word for me. Like as a French person. Like in in French it would be a length scale. So what is a length scale, Bill? Yeah.
speaker-1: Yeah. Yeah, so a lot of kernels that we use a lot have a parameter in it called the length scale. ⁓ there's all kinds of kernels that don't have length scale parameters, they have other parameters. There's useful kernels that don't have any parameters, you just say they're similar and you go from there. ⁓ there's kernels over graphs, there's kernels over trees, there's kernels over all kinds of weird data things, but working in just sort of regular data where if x is five and another x is ten, the length scale parameter says, okay, is five and ten, they're five apart. Is are we going to think of that as similar or not? You know? And the length scale, what it basically does is scale the inputs. That's the simplest way to think about it. And you can do a little do a little algebra on a kernel function like the exponential quadratic, and you can see that the length scale just defines just divides each x by a scale. And you know you can divide x equals five and x equals ten, you know, by divide them both by five and now they're one and two. So now they're only one unit apart. And then you go, okay, well that's pretty similar if the length scale is five. But if your length scale is like a hundred, then five and ten are hardly different, right? That's basically the same number. ⁓ and if your length scale is like point zero zero one, then five and ten are really, really far apart. And those are Hardly the same thing at all. And so your y if you have X is five and another X is ten, then your Y values there shouldn't be related to each other at all. They they have n really nothing to do with each other. And so that's that's what a length scale is doing in in a lot of the common kernels like the ones from the Matern family and the exponentiated quadratic that you'll that you'll see if you look at the Prime C examples or or most examples with GPs. Yeah.
speaker-0: Mm-hmm. Yeah, and actually so you'll tell me how wrong that is, but usually when I teach GPs, a way I help students understand about length scale and amplitude is I use the first I think the most probably the most useful function of the whole PyMC package is the GP plot function that I think you authored during your G Soc, right? This plot that you get with the envelopes and like first the colors are amazing with GPs because they are so, you know, like uniform and continuous. It's just always extremely beautiful and pleasing.
speaker-1: Make pretty plots. That's that's honestly another reason.
speaker-0: They make amazing plots. Yeah, yeah. And you can use and you know you can use the best color palettes from Matplotlib, like you the VRDs and CDs and Magma and they work. Whereas usually if it's like if you've got categorical data as I have all the time, it's such boring palettes, you know, it's just like, ⁓ you know, it's just one discrete color after the other. It's like, ⁓ my god. GPs are great because continuous. So color palettes. That function is amazing. Because you can see really the GP if it's to D of course. And then so thanks to that, once you have that plot, basically the X axis is the length scale. And it's gonna like the length scale is basically gonna tell you how far do the relation go on the x-axis. Yes. ⁓ and it's like kind of a memory process. ⁓ the memory of the process, because like if it's a long memory, then the length scale is gonna be long. If it's a short memory, short length scale. And also sometimes the length scale can be ⁓ the memory can be spiky. It can be like if it's periodic, or actually every quarter you remember about something, you know. It's like ⁓ or like every year, you know, ⁓ shoot, I have to do my taxes, you know. So it's like that's like the the annual parody taxes, parody kernel, you know. so yeah, that's the length scale. And then y axis gives you the amplitude. The amplitude is basically a Scanning factor, and as you were saying, if you look at the most of the math of the kernels, the amplitude is just like, you know, that small factor just on the left of the kernel ⁓ formula. And that basically tells you how far can the functions go on the y axis. If it's a high amplitude, then well it's gonna it can go the GP can be very, very ⁓ capricious, let's say. If it's a small amplitude, it's like it's gonna be a very calm. Very calm T P, you know, like it's not gonna be
speaker-1: Yes. Yeah, I know. That's that's exactly right. It's the same thing because when you're thinking of those one D ones, you can think about a time series. You could say, okay, and and it's like a memory, like you're saying, right? So X is five hours and X is ten hours now, right? And if your length scale is really short, then it's like, well, it doesn't know anything. Like ten hours is nothing like five hours. So much has changed in that amount of time. So we don't know anything like what happens at five hours has nothing to do at ten hours. And so the same thing if X if the length scale is a hundred, then if you're talking on scales of hundreds of hours, then five and ten are basically the same time. And so then you're gonna have something smooth if you're if you're thinking about things in that scale. So it's it's the same that it's really funny you mentioned that function because just to go down the cause I was I just kind of hid that in there to help make plots for the documentation. Make pretty plots. But it has been useful and I use it all the time. Even just to plot one D stuff 'cause you it's like a nice shaved thing, but it it's funny, it has a horrible syntax and I just stuck it in there to be handy for plotting because I didn't want to have to like
speaker-2: Have you heard of have you heard of Hiram's Law?
speaker-1: No, what's that?
speaker-2: It's a law that that any any piece of your API, if you're a programmer, if you have anything, no matter how hidden it is, no matter how private it is, if it's out there for long enough, somebody will take it and make it an essential part of what they do and you must maintain it.
speaker-1: Yeah. ⁓
speaker-0: Yeah, eighteen. I don't know because I mean that that you know, that filters users, you know. Because if you make it easier to use then more users are gonna use it. And then you need to maintain it actually. Whereas if it's an Easter egg, you know like Yeah. If it's like, you know, animal style at In and Out Burger, you know, it's like just just the people who know knows it's there. You know, it's like it's fine, like you don't need too much maintenance. It's like you know like the the people who use it, that Easter egg, they you know, they know their stuff. So you could you could use that, you know.
speaker-1: Yeah. Just hidden deep in G P you've been
speaker-0: Exactly, yeah. C GPU tools. Yeah, but they use it all the time for like, you know, even for regressions and and stuff like that. It's amazing. Yeah.
speaker-1: Yeah. No, it's ⁓ I've tried to make it nice a couple of times and then it gets too complicated and I get stuck because I'm like, Well, this option or that option and now I have to actually make it good and then and then I get a little overwhelmed and have to do things that are whatever, do immediately, you know? Mm-hmm.
speaker-0: Yeah. JC, you wanted to add something.
speaker-2: I wanted to add that it's incredible that you've been on the West Coast for like two, three months and you're already preaching the gospel of in and out. Like you're really you're assimilating.
speaker-0: I do, I do. Yeah. So I mean so the fun story is that I already knew about In and Out because I was in Stanford in twenty twelve. And so that's where I discovered, you know, the truth. ⁓ so yeah, that was a fun that was a fun three month like I discovered in the same trip, In and Out, Jack in the Box, which is amazing when you're a student and you don't have money and you're hungry at two AM. ⁓ And and I discovered Netflix. That was the very beginning of Netflix streaming on your computer. And in France at the time it was the middle ages. We had like one big actor who of course didn't want that because they were selling forty Euros a month ⁓ ser premium service so that you would get that only on your TV and not even on demand. So so then when I arrived here, I was like, ⁓ my god. And I binge watched all the seasons of ⁓ twenty four, which was amazing. ⁓
speaker-1: Nice.
speaker-2: Yeah, I guess I guess the mini ted couldn't do streaming at that time. No, no, no, no. ⁓
speaker-0: But it's just you know, not not ⁓ it was the law, you know. I mean we like that in France, so
speaker-1: Nice. But in fact, I happen to live one mile away from the last blockbuster.
speaker-0: ⁓ yeah? ⁓ wow. Damn. And they're there sometime?
speaker-1: Yeah. I still haven't been in I have not been in because I do not have a VHS or D V D player, I will admit, but I need to. 'Cause I I mean, yeah. Streaming services are terrible now. I honestly would think it'd be nice to just go rent a movie. Yeah. Not like which serve ⁓ doesn't have it. Yeah, I have to unsubscribe from this one and resubscribe to that one. It's done.
speaker-0: Yeah, none's gone. The golden age is gone. Yeah, yeah, yeah. Now it's like now you actually need to pay like about one hundred bucks for your each month's food ticket.
speaker-1: Dumb terrible. I'd rather go just rent the one movie or series that you're gonna watch. Walk over and get a D V D and be a lot easier. Just the one. Return it.
speaker-0: So I actually who should ⁓ like you know, in in the in the biopic they'll do about you and Gaussian processes, ⁓ what b who should play your role, Bill?
speaker-1: Ha ha I didn't prep for that question.
speaker-0: I'll give you a few minutes then we'll we'll we'll come back to that. Yeah. ⁓
speaker-1: I ⁓ I always think about that. That's a
speaker-2: I see Ryan Ryan Reynolds, you know, you're like a lovable, like lovable loser type, right? And like
speaker-1: You find it.
speaker-2: They came out mean. I didn't mean for that to be mean. I meant for
speaker-1: I just heard lovable. That's all I heard. Yeah. I've gotten I've gotten Walmart Woody Harrelson one time. And that could be mean too, but that could also be positive because
speaker-0: Yeah, me too. Yeah.
speaker-2: Hear me out th hear me out though. It's Ryan Reynolds and the name of the movie is The Notebook. The Jupiter Notebook. Did you
speaker-1: Nice.
speaker-0: Turn out boo. ⁓ that's great. I I I'd buy I'd I'd go watch that, yeah for sure. Invite us to the to the premiere, please. ⁓
speaker-1: That'd be good. That'd be good. I'll ⁓ I'll prompt the AI to just generate that for the whole world. It'll be done in an hour. There you go.
speaker-0: So actually so to get back to get back to business, ⁓ why why did you use PyTensor for this new ⁓ this new package? You know, like what what is that adding what's the difference with the vanilla and HSGP sub module from PyMC? What would users even be interested in in that?
speaker-1: Because PyTensor is awesome. The end. No, but like it is though. It's awesome. And ⁓ One of the things so Pi PyTensor is like a you know, it's like a it's a very boutique hipster deep learning framework with auto diff. You know, nobody knows about it. It's not one of the big mainstream TensorFlow, Jack's, whatever, Torch, PyTorch, one of those ones. But it's like way better, actually. Because okay, Jesse should answer this one, but I'm gonna I'm gonna Bumble over a couple reasons why I think it's awesome is that you can if say, okay, say all those big companies decide to never support the big autodiff stuff again. And there's this new cool one that everybody starts using. And then everybody goes, ⁓ we gotta rewrite all of our code. And PyTensor, you can just wrap it. And now it's a back end. And PyTensor's still alive, right? So PyTensor is what Theano has kind of grown into over the years. And Theano is super old. Like how old is it? Thirty years? Twenty years? Thirty?
speaker-2: Two thousand two, two thousand three. Like the early commits aren't even on Git it predates GitHub, so the early commits are lost to time.
speaker-1: Amazing. Wow. Yeah. So that's it's and it's still around. It's still kicking because of this kind of ⁓ design choice where they had a C backend, but you could run other stuff. And so as long as you can kind of hook into it and run it from Python, you can run anything. So it's gonna be around. ⁓ and it also has this cool so okay, Jesse should take over at any moment I start saying things that are wrong.
speaker-0: Damn.
speaker-1: But like it makes a graph of your compute and it represents it in Python. Do you like I want to do one plus A plus B. It says, ⁓ there's A and there's a plus and there's a B. And then and if you do like A, I want to do A plus B divided by B, PyTensor can look at that and go, why don't you you just want to do A plus one? That's what that is. We don't have to actually divide B by B. So we'll scratch that out and your actual program is just A plus one. So it does these optimizations for you. So it kind has this little compiler step. And this graph it makes, you can work with it. So it's in Python. And it's not some deep C level complicated thing that you don't really have access to. It's just in Python. And that opens up a lot of things you can do with it too. So those are the two main things. And both of them are really handy for making ⁓ GPs.
speaker-0: Huh. But so that would be on the back end, right? Yeah. users would not see that. So
speaker-1: Well how unless they want to. Yeah, they might. Yeah.
speaker-0: ⁓ yeah, no, for sure. Then what do they get? Like, do you have actually examples already of of PTGP that you can show here or that we can put in in the show notes?
speaker-1: Yeah, ⁓ for sure. Yeah. We've got some stuff we could show. ⁓
speaker-0: That'd be great. So while you get while you will you get set up ⁓ you have that already? Just
speaker-2: Yeah, yeah, yeah. I'm all I'm all ready to go. I'm all ready to go. I'll see what I want.
speaker-0: Take it away and actually I'd I'd love to hear you, Jesse, about ⁓ the new stuff you were hinting about on PyTensor that made
speaker-2: I'm so glad you asked. What a what a coincidence.
speaker-1: Yes, there's a lot more cool stuff that PyTensor can do.
speaker-2: So ⁓ let me share my screen. Can you see this?
speaker-0: Awesome, yeah, yeah, yeah. Yeah yeah, so we have ⁓ Jupiter notebook, not yet ⁓ Bill's bio biopic, but ⁓ almost.
speaker-2: I need to I need to I need to correct
speaker-0: So if you're listening I I recommend switching to ⁓ switching to YouTube.
speaker-2: And like and subscribe on YouTube. That helps the show. True. The So I need to correct the record before I get into the Jupiter notebook because I made a I made a joke about who would play Bill and I got the actor wrong, right? It's Ryan Gosling is the heart throb with the abs who is in Jupy who is in
speaker-0: In the notebook.
speaker-2: Reynolds played. Which is Bill. I think it was a very good joke, but I got the actor wrong and I want the show to reflect that.
speaker-1: The girl?
speaker-0: Thank you. This is a show about ⁓ about accuracy, so thank you so much, yeah. I think you can
speaker-1: Yeah. The abs are the abs are spot on, so that part's
speaker-0: I think you can zoom in a bit actually, ⁓ Jesse.
speaker-2: Yeah. Yes, yes, yes. If I can, how does one work? Let's see. Command shift. Yeah. That's right. We gotta get the shift in there. That was what I was mucking up. So I'm going to jump to the the good stuff here, which is kind of how does this all work with a GP? So we've talked about GPs are math heavy, they're hard to work with ⁓ because they're expensive. Why are they expensive?
speaker-0: Come in place.
speaker-2: So I have on the screen kind of the two main equations for a GP. One defines what this kernel matrix is, and the kernel matrix is just a function of an input matrix X and it's pushed through some function. Okay. The function has to have special properties. Essentially it ensures that no matter what you give it, what comes out Hey Bruno. No matter what you give it, what comes out will be a valid covariance matrix. And then you're going to take this valid covariance matrix and you're gonna pass it into ⁓ the log p of a multivariate normal. And the log p of a multivariate normal has a really unfortunate property that you need to invert the covariance matrix. So this first term here for the listeners is y transpose covariance inverse y. So this inverse step is actually like O n cubed, a little bit less than O n cubed, like O N two point seven or whatever. It's really bad. And it's also quadratic blow up in memory. So if you have ten thousand data points, that means you need to store a ten thousand by ten thousand matrix in your computer. That's not so bad. But if you have a million data points, like you're kind of lost. So what can be done? This is I'll say one more thing, that the way that I've written this here with this actual k to the minus one is kind of the math way of thinking about it, where you say, I'm gonna actually invert this thing and then I'm gonna do standard matrix multiplications. ⁓ I say that evocatively because in a moment I'm going to make a pitch for why PyTensor's rewrite system is actually nice, even if it's on the back end. So what can be done here? If you're if you're sort of hip and you know numerical computation, you kind of know that k inverse y is a solve operation, and that you shouldn't do inverses, you should change this to be a solve. Right? So you can write this in code as solve ky. And then if you're really, really hip, you kind of know that, well, k is positive definite. And so I could do a Cholesky decomposition on K. I could store this. decomposed matrix. And then I could use a triangular solve or two triangular solves to get this y transpose k inverse y. And that's a little bit cheaper. And then all the way for free, we haven't talked about this log determinant of k. But once you have a Cholesky decomposed matrix, the Chalesky matrix is always lower triangular. And the determinant of a lower triangular matrix, as everybody of course knows, is just the product of the diagonal. So now you're not doing any kind of fancy Linear algebra, you're never calling out to Lopak because you already took the time to make this K, this Chelesky factor of K, you could just read off the diagonal and do standard math, ⁓ standard arithmetic. So GP libraries, what they'll concern themselves with is sitting down and thinking really hard about how do I write this thing out in the most computationally efficient way possible, and then stuffing it deep into the bowels of their library. And hoping that nobody ever looks at it because it's not going to be obvious at all if you just came from a machine learning textbook that showed you this equation, how it's actually going to be written once you go to the code. What's nice about PyTensor is what I'm going to do is I'm just going to write it exactly as it appears in the textbook. So I'm going to say that this quadratic term is just y Pt Linaugh inverse covariance y. And this is like absolutely verboten because now you're directly doing this O n cubed inverse. You fool. You've left money, you left cash on the table. And then log debt. I'll use the s log debt function which computes the log determinant of a matrix. And then I'll calculate exactly the formula that was shown above. And I'm eliding details about the X quad thing here and and and whatnot. And you could do that. Okay, now PyTensor has this suite of rewrites. Anytime you give it a program, it's going to look at the program and say, what do I know about the relationships between the objects in this program such that I can make it a little bit better, a little bit more stable, a little bit faster. In this case, ⁓ I have this counts linalg ops function, and it's going to tell us how many linear algebra operations are in here and what they are. And in this case, we end up with one s logdet function and one solve. So we didn't write a solve, but we got a solve. So already we've won. That's nice. Now what's new is that we have this assume function in PyTensor. And what assume does is it lets you tell PyTensor that certain objects in your program have certain algebraic properties that aren't that are not necessarily inferrable from just looking at the operations. In this case, we can tell it look, k is a covariance matrix, a covariance matrix is positive definite, and a covariance matrix is symmetric. It's going to remember those things and it's also going to reason logically about this over a lattice, like a logical lattice. And it's going to say, well, if I have a positive definite matrix and I take a dot product with another matrix, well, what do I know about that other matrix? Well, it's positive definite two. So that's going to stay. Is that true? Are you shaking your head? Did I get that wrong, Bill? Is the dot of two positive definite? It might be wrong. But the point is
speaker-1: No, I was I was you're good. Yeah.
speaker-2: Okay, okay. ⁓ the point is is if it knows something about another matrix, it'll propagate these properties forward. Here's a more simple example. I am a diagonal matrix, and I tell PyTensor you are diagonal, and I multiply it by some scalar. You know, I take i times four. Like while the zeros are all going to stay zero, the i is going to become a diagonal of fours, the result of that is just ⁓ a diagonal matrix again. And actually PyTensor knows. That, ⁓ if you're diagonal, I don't even need to look at the off diagonal elements. I'm just gonna take that diagonal, which could just be 10 numbers instead of 100 by 100, I'll do 10 multiplications instead of 100, I'll give you that 90% savings. So if I tell it that K is positive definite, and then I run the exact same ⁓ compile, then what I come out with is one Chalesky decomposition and one Chalesky solve. So you see the determinant is completely gone. That's become just a summation because we're doing the log of a product. So I just add the diagonal. And then we're doing one Chalesky solve to ⁓ actually do the ⁓ the solve got further specialized, is what I want to say. So more numerically stable, a little bit faster, really great. And everything like numerically is completely equivalent. This notebook will be in the show notes. We're gonna get it live on PyTensor. ⁓ examples gallery. So there's a bit more and it shows you kind of how you can extend it. And Bill actually did more specialized things. So he actually specialized the inverse function. We in PyTensor we specialized solve, but we didn't specialize inverse. So if you took the gradients of this, you would be un un unfortunately there would still be a matrix inverse left over and that's kind of a bummer. Bill said like, well let's just change that into another Chalesky solve. And we can reuse the Chalesky factor we already have laying around. So you don't even need to decompose it again. And then it makes it even better. You end up with four Chalesky solves and one decomposition. So very hackable, very extendable. That's kind of the power of PyTensor. and in this case, it gives us three things. It gives us numerical stability, it gives us readability and kind of maintainability. ⁓ and then the third, and I guess I could just have mixed it into the second, but three is a wonky number. It's approachable. Right. So again, if you're someone who's coming from a textbook and you want to understand how your code works, you go into the library, if you go into PTGP, the point is that you're going to see the closest thing to how it's written in the papers. And then we're going to lean really heavily on PyTensor to give us that same speed up that the g that the fellas in GP JAX and GPflow and get ⁓ from handcrafting these these functions really carefully.
speaker-1: Yeah, and that opens up a lot of stuff, right? Because a lot of these, like a Cholesky, there can be certain structure in the matrices and you want s different solves and and all of that's now on the table. But I want to just kind of highlight, Jesse, you went over something really quickly that I think is really cool, is that you wrote a a rewrite that replaces the inverse of A with a Cholesky solve when A is known positive definite and it was eight lines of code and you did not have to add it to PyTensor. It's just sitting in your notebook. And then now it got applied to the rewrites that PyTensor's doing. So and this calculation that you ran, is that using there's a Jax backend, there's a Numba backend, there's a C backend. Isn't there a what's the Mac thing? MLX? Right. Someone's finishing that up.
speaker-2: Yep, there's an MLS and there's also a PyTorch. So we have PyTorch. Nice and I mean
speaker-1: Yep. So you could run that in all of those just by changing a flag. you know, if whatever a certain a certain tensor library is not maintained anymore and there's a cool new one, you can use that one with PyTensor. And so if you're working on some library and you there's like, ⁓ there's a rewrite that's just specific to this thing I'm doing, you can just have it in your library without adding it to PyTensor. Or you can add it to PyTensor, one of the two. It doesn't whatever makes the most sense. And you know, you can just keep going with what you're doing without PRing it in and do that on the side and it's just it's a great library to work with.
speaker-0: And so thanks a lot, Jesse. Yeah, that's I think it's great to see it actually on the screen. ⁓ is there already a Python a doc ⁓ website for PTGP that we can link to? Awesome. Yeah. So I'll do that. Also link to to the examples. To make this even more concrete for listeners, if you if they wanna use PTGP, so they install it with Conda or ⁓ Peep. Does that depend on Pint C or is that just Pytensor?
speaker-2: I'm so glad you asked. Let's quickly look at what a PTGP model actually looks like.
speaker-0: Yeah, perfect. Yeah. I think it's great if we show basically yeah what what that looks like, how they would even write a ⁓ a P T G P model.
speaker-2: So the answer to your question is yes. PTGP depends on both or builds on top of both PyMC and PyTensor. And the decision was made very early on, like kind of right away, to say, what what is PTGP not? It is not a probabilistic programming language. And maybe Bill, you can talk about kind of your thinking there because it's I think interesting and important. But the point is that when you need a prior, When you need to simulate, when you need to sample, we're going to reach for a PyMC. When you need to do something with GPs, some specific GP logic, we're going to have functionality in PTGP that does what you need to do. So if you look ⁓ here, I have a model, a very simple model. ⁓ we're fitting on this motorcycle, the Silverman motorcycle data set, which is recording like the head velocity of a crash test dummy when ⁓ it impacts, yeah. So this is kind of the case that was being discussed in the introduction where there could be a physical model here. There's some Newtonian model of physics that describes how the evolution works, but we're gonna say we're not so interested in that. We're interested in understanding, maybe we're interested in like interpolating or we're interested in ⁓ extrapolating, or we're just interested in something else, and we want to kind of take out the physics piece of it and then study some other effect that comes in on top. And so the G P is gonna suck that up and then we can do whatever we So the actual model definition is just nine lines of code. We're going to define three priors for the length scale, for the amplitude, and for the observation noise. And those will just be normal PyMC priors. And then we're going to take the PyMC prior and we're going to multiply it by a P a PTGP MATERN52 kernel. And you can see kind of the the watchmaker's hand behind the design here. Because the PTGP kernels look very much like the PyMC kernels. So if you're coming from PyMC, this should all feel extremely familiar. And then
speaker-1: Like other libraries kernels like G Py or GPflow, I mean they all kinda use the same design for good reason. And yeah.
speaker-2: Absolutely. And then the the magic of PTGP is kind of this line here where you're going to specify the approximation that you want to use to estimate your GP. And I use this term approximation loosely because in this case our approximation is no approximation. We're going to fit the exact full GP. But w in the introduction I used the term opinionated and I used I described PTGP as the the kind of blue collar, you know, yeoman's practitioners ⁓ GPU library. Yeah, I I dressed for the occasion. I I put on a shirt just for that joke and I waited an hour for it.
speaker-1: ⁓ huh.
speaker-0: Well done. That's that's dedication.
speaker-2: The consequence of that is that what is provided are battle tested algorithms that scale up with good guidance on when you should use out approximation one versus approximation two versus exact. In this case, we have a very small data set. Everything's copacetic, so we can just use the exact thing and we can get an answer. The way you get an answer is you just call PT GP dot fit. Bada bing, bada boom. You're done?
speaker-0: Okay, and so that that fit stuff is what does it do, is that ⁓ L E thing or is that ⁓ C ⁓ C?
speaker-2: Bill, you want to talk about the philosophy of fitting?
speaker-1: Yeah, yeah, sure. ⁓ like I kind of said earlier, like there's the GPs that kind of make sense within a probabilistic programming language. So there you're gonna use MCMC, and then there's GPs that are sort of like a GP model, and this is models a GP. And that's kind of what's going on here. And in this case, we're doing map estimation, right? Because we've put priors on the length scale and the amplitude eta and the noise sigma. Like we've got those priors thanks to PyMC. That's already done. And so we're gonna run probably I think L L B F G S L F B or whatever, that one on ⁓ and fit is like a nice quick wrapper over the sort of default basic, just optimize this thing ⁓ using sci pies minimize. And ⁓ yeah, it's ⁓ you can kind of It's set up so that you can kinda get in there and tweak the optimizer because you can run into weird edge cases. Like Alex said, fitting GPs is hard and you gotta make it easy, but this is the easy code to just try it out and run it. Run the minimizer and go, you know, when you're in a a pretty straightforward situation.
speaker-0: Nice.
speaker-2: Although and although it is map, so for the kind of good Bayesians in the audience, you will know that map is sort of poo-pooed because it doesn't give you posteriors over the hyperparameters of the model. And what that often ends up hap what is what ends up happening is that you don't get your posterior predictive distribution, you're just point estimates all the way down. And so you took this beautiful probabilistic model that you could very well have fit in Perfect Bayesian form and just reduced it to Dare I say a frequentest result? ⁓
speaker-0: I'm gonna have to edit that part.
speaker-2: Note but just beep it.
speaker-1: We dropped that. ⁓
speaker-0: Yeah, I'll beep eat.
speaker-2: ⁓ Now what's what's about ⁓ Bit actually be like
speaker-0: It'd be like an editorial choice.
speaker-2: What's what's cool about GPs and what they have in common with state space models is that they're kind of Bayesian from first principles. And so even if you only have plane estimates for your parameters, you still get distributions out for your estimates. And so you can see here that we have plus or minus two standard deviations, everything's normal, so that works out to your ninety-six point whatever ⁓ percent HDIs. ⁓ and yeah, it's it's it's nice. You get a good result, it's fast. It's simple. We like
speaker-0: And so it's that Bill because I remember you explaining that to me ⁓ a few months ago or years ago where ⁓ you were like, ⁓ no, you'll you're still gonna get uncertainty estimation wave map. I was like, What? What water is mocking? ⁓ and you're like, No no, you get that because if I remember correctly, it's because it's ⁓ it's an empty normal, a multivite normal, and so thanks to that basically you get that. Uncertainty estimation. ⁓
speaker-1: Right. Yes, that's right. ⁓ because you're it's a Gaussian process, right? So you've got this kernel. The things you don't know are like the length scale and the amplitude. So those make your kernel, so that's a matrix, and then that matrix it's ⁓ it's in a big multivariate normal. So then you're drawing samples and then you have the uncertainty coming from that. And yeah, you are you are underestimating the uncertainty a little bit because There is actually uncertainty in those hyperparameters. But if you if you were to actually do it that way and use MCMC, you slow way down because you need to make a new matrix every time you change the hyperparameter and make a whole new matrix, and then do a whole new Cholesky and a whole new whatever to get one sample and repeat that every time. So by using map estimation, you don't have to do, you can do the Cholesky once on your map estimate and then draw lots of samples from the multivariate normal Instead of doing the Chelesky every time to just get one sample.
speaker-0: Yeah. Yeah, so it's like it's slightly I mean it's underestimating uncertainty for sure. In some cases it it will be slight, in some cases it will be high, so it will depend on the use case. and and yeah that's that's a choice. But what's the like is there a default way of sampling in PTGP? Is that is that the fit method or like people can just choose whatever they want?
speaker-2: So, and maybe Bill, you can disagree with me here, but essentially what is recommended is that you look at kind of the size of your data. And if you have very small data sets, we're talking hundreds of data points, then you can just go for the unapproximated. There should be no issue if you have a modern computer, you can easily solve it. There's no reason to cheat yourself of the full thing. As you scale up to something in the order of thousands to like five thousand, six thousand, there's this very nice Variational free energy approximation, which is ⁓ an induce it's a part of this inducing point family, and we're going to need to maybe talk a little bit about inducing points. I hope we have time. But this is kind of the most exact inducing point formula. Bill, if you agree with that, please.
speaker-1: Yeah. It's inducing points when you have larger data sets, larger, not large jest, and you also have a normal likelihood. You need that too. So there's a lot of these that's why GPs are hard. There's a lot of these like if this, if this, if this, then this is the best and and it's it's tricky to navigate that.
speaker-2: And if you have really huge data, just from say five to six thousand to infinity, then you're going to reach for an algorithm called ⁓ SSGP, S V G P I'm sorry, stochastic variational GPs, ⁓ because that can do mini batching and you can just let it rock with an atom optimizer tuned to whatever batch size your computer can handle, ⁓ so it sort of scales infinitely. ⁓ at least that's the promise.
speaker-1: Hopefully. Yeah. But no, it it scales up for sure. But yeah, yeah, yeah.
speaker-2: And there's there's more exotic and exciting tools coming. Things like you can take infinite data points and sort of project them onto a sphere somehow, and then that turns it into an O one problem. So no matter how big your data is, there's these exciting new approximation methods where the inducing points are picked in such a clever way that it scales constantly with your data. And then Everything's really, really fast.
speaker-1: Yeah, it's kinda trippy. The inducing points don't have to live in the space of your data. They can actually live in some other space. And in that other space, maybe it's more efficient to do linear algebra. So Yeah.
speaker-0: Yeah, I mean that makes sense. Like bas in inducing points the idea is like you try to summarize your your data set, right? So it's like I'm guessing does it have some familiarity with ⁓ PCA or stuff like that?
speaker-1: ⁓ Kinda. It's easier I kinda think of them as like spline knots. Sorry, Jesse, go ahead.
speaker-0: ⁓ right.
speaker-2: I was just I I was gonna push back on PCA just because PCA summarizes the column, the columnar dimension. So you have like a thousand features and I just wanna find so I wanna find ten features that summarizes it really well, but I still have as many rows as I have, right? Mm-hmm. The important thing about the the the most important thing about inducing points is that you say, I have too many rows. Like I don't need all this rows. I can I can get the same answer with many fewer rows. I just wanna carefully pick The synthetic rows I'm going to use to solve my problem.
speaker-0: So it's just like sampling. This is like polling.
speaker-1: Little. Yeah.
speaker-0: You have a full population and you're trying to find a the smallest representative sample unbiased of your population to learn the actual ⁓ election result.
speaker-1: It's exactly like that, yes. And the inducing points don't have to even be draws from that population. They can be c some kind of thing in that space or another. Yes. But they summarize it, yeah. They reduce the rows. Yeah.
speaker-0: They could be they could be like two voters or three voters like summarized and like d s you know, aggregated into each other. That'd be a weird poll, but I'd love to see that.
speaker-1: That's kind of what you think about if you're like, can I use inducing points here? You're like, is there kind of like a repetition in my data? Do I have lots of is my data dense in certain places? And you you want to keep it, you don't want to throw it out because you don't really know where it's dense, but you maybe have some sense that it's a bit too dense, you know? And it's like this function that Jesse's got showing on the screen is like the data's denser than what you would need to play connect the dots through the curve.
speaker-0: Curve.
speaker-2: I have a kind of more extreme example as well. This is the daily the daily monoloa data set, it's quite famous data set. And this is daily data going from the present all the way back to nineteen seventy. So we're talking about fifteen thousand data points. But if you see, if you zoom in, I mean the data is very highly structured. Right. And like you simply don't need all of this. Now I picked six hundred inducing points. So if you look at the bottom of the graph, and and for the viewers I have the monoloa data set which goes up into the right and has periodicity. It sort of wiggles up and down. And on the bottom I've plotted the the places where I've put down inducing points. And I put down a ton of inducing points. It looks extremely dense, but it's still like less than one tenth of the actual data. So even with this kind of extreme example, like I didn't bother to do anything smart, right? I just said, I've got this time series data. I'm just going to lay down a bunch of dots on the time series kind of evenly spaced. And then you get the same answer you would have gotten if you solved the whole thing. Because there's just too much there's too much re ⁓ repeating information in the data set.
speaker-0: Yeah, that data is amazing. I I'd love to get that data. I've never feeded I mean s feet or sample the G P on on that that data. It's incredible.
speaker-1: Yeah, it's a good example. Classic example.
speaker-0: Yeah. Yeah. And so all the all the methods to infer the GP parameters here that we've talked about, this is all already available in Pt GP, right? So feed, sample I'm guessing with ⁓ with classic nuts, and S V G P things like that.
speaker-1: Yeah, except for nuts because they're sort of we're we're still thinking about where to kind of split the line. Because that's that's a that's gonna be you're using that within PyMC and PTGP sits on Py ⁓ C It's a it's a fuzzy line. But yeah, there's gonna be no there's definitely not going to be like a different implementation of nuts that PTGP would use. You'd use NutPie or the sampler built into Py ⁓ C and maybe you'd use some PTGB code to construct the the GP itself or or not and then and then you'd use PyMC for the sampling for that one. But yeah, the the optimization stuff though is going to be in PTGP because one of the you know one of the goals I have is to make it easy to use, right? 'Cause ⁓ it's There's a lot of great GP libraries out there, and they kind of occupy different niches. And a lot of them implement, you know, the method from this paper or that paper, but there's not a lot of guidance on past the basics on when to use what and also how to fix issues when they come up. So often you can fit something and run into problems, or the fit will do this and that. is a sign of this pathology and this is what's wrong with your model and you need to fit it. And it's kind of like how how nuts is where you look at the trace and it says something about identifiability or something like that in your model. and it's a little simpler though because you're mostly doing optimization stuff. But ⁓ I just want to make we want to make sure that how to fix that, how to kind of cycle through models quickly because we all work under time constraints and we got to get stuff done. ⁓ so how all of that is ⁓ how how you can it's made with that in mind so that that's easy to do, you know? So you're not in that situation where you're like, well, that looks like a cool paper. Maybe I could try it. It'll take me a really long time to implement and then it doesn't work, and then is it because I did something wrong, or is it because this method is wrong and what do I do next? And ⁓ so sort of have the batteries included and the the help kind of included.
speaker-0: Yeah, yeah. Yeah, I love that. and G Jesse, you were were you done with the demonstration or did you have anything else to show?
speaker-2: Well, s we can do one more. I didn't want to ⁓
speaker-0: Yeah, yeah. Let's let's think it's great to so that people can see basically what they can do with that.
speaker-2: Hundred percent, yeah. So let me just point out so we're looking at the VFE approximation. This is the one that is for a medium amount of data points. I think I I pushed it pretty hard with this Monoloa data set. It's fifteen fifteen thousand data points. So it took about nine minutes on my my MacBook to fit, which is a bit longer than one would hope, but it it works. I I wanted to point out one thing Bill mentioned. That it's batteries included. So we saw this fit function, but everything is also totally decomposable. So there's this compile scipy objective function and it gives you back all of the bits that you need to then go ahead and call minimize yourself. So if you wish, you can ⁓ customize this as much as you want. If you love genetic algorithms, you know, you can go into the differential evolution thing in SciPy and you can give it your function and you can let it rock. Right. So ⁓ we do LBFGSB because this is a really good algorithm that's hard to beat, but we also expose all of the components. So if you're a user who's running into problems and you think that you can do a bit of tuning, that that's all available for you. Nothing ultimately is hidden from you.
speaker-1: And oftentimes, even in this situation, if it's if you're having something that can happen, kind of a failure mode, I guess, of this VFE approximation, is that you'll have problems optimizing both the inducing point locations, the noise, and the hyperparameters at the same time. Sometimes ⁓ and you'll you'll just see training. Do something weird. I forget, either sigma goes high or sigma goes low, because it'll be like there's no noise or there's all noise. And then but the fix there can be to first optimize the inducing point locations and alternate between that and the hyperparameters. And sometimes you want to hold sigma fixed for a while, optimize those other two things, and then let sigma start optimizing so it doesn't run off to one of those one of those bad spots. And so It's that having this all broken up lets you do that. And then there's there's like skill files that we're making and the skill files for VFE are in there. So that when you're running this and you're using an LLM like we all do now and it sees this and there's a problem with optimization, it can look through the skill file that comes with it and be like, Well, this happened. This is how you fix it. And it just knows how to do it immediately. And so because there's a few common modes, right? And so You can either, you know, you can start with dot fit, pg dot fit and just do it, or break it down and be able to hold different parameters fixed or not. And yeah, just being able to do that quickly and then also being able to hop from V F E to some other approximation and change your priors kind of quickly, just so that the workflow of actually using these things goes smoothly for you. Mm-hmm. That's kind of what we're going for.
speaker-0: Yeah yeah. Yeah, I I love that. And since you've opened the kind of worms of agents in AI, that's great. So I'm gonna I'm gonna play us out ⁓ with that. Well done, Bill. ⁓ but so first thanks all Jesse for this ⁓ Jupyter notebook and and you've proven once again that ⁓ you're such a boomer because ⁓ you're still using Jupyter notebooks. Like that it's not even Jupyter Lab.
speaker-1: Yes.
speaker-0: It's like Jupiter notebooks, like twenty tens vibe. ⁓ in the age of AI agents, that's ⁓ well well well done, ⁓ I appreciate
speaker-1: With your ⁓ C P server for Jupiter.
speaker-2: I did. I wrote my own I wrote my own Jupyter classic MPC ⁓ C P server. So if anybody else wants to boom around with me, Jesse Grobowski slash ⁓ NB NB ⁓ C P bridge, ⁓ it installs the server and it installs the bridge and you can collaborate with ⁓ with an agent from your cloud code terminal and everything lands right in your Jupyter classic session.
speaker-0: Hmm, nice. How how is your experience with the Jupyter notebooks and and AI agents?
speaker-2: So Bill is the one who turned me on, so that came out that came out weird. You wanna believe something? Believe that.
speaker-0: Yeah, I bet ⁓ I mean we're still talking about Jupyter notebooks.
speaker-2: Someone who turned me on to AI and Jupyter Notebooks and there was a blog post from I think the the author of NB Convert, like I got an email like a day later and he was saying there's a stack you can use and you can plug there's a there's an MCP bridge for doing like collaboration, like multiplayer notebooks, and then you're gonna plug in your agent and then the agent can work in the notebook. And I tried it out and it was just miserable and it was just terrible. I hate Jupyter Lab. I don't know how anybody can use it. Like you run a cell and it jumps all the way to the bottom, and then you gotta scroll back up to see what the output is. Like by default, it doesn't have like comp like closing parentheses automatically. You can't do the like question mark to get the like little pop-up at the bottom that shows you the health. Like maybe, maybe I'm absolutely like Stockholm syndrome, but it just seems like so much worse than like the old school like bare bones HTML thing. So I said, you know what, I've got a I've got an agent, I've got a Claude Max account. Give me ⁓ give me an MCP server for this. And it like one shot it and it was really good. And I've been sort of tinkering with it and improving it ever since. ⁓ I haven't had any problems with dropping. I haven't had any problems with like clobbering each other's stuff. There's other problems, right? Like it's janky. It's the same as everybody's hobbyist AI but it's like my janky software, so I'm willing to overlook the issues. I think if somebody else does it, they'll say like this is crap.
speaker-0: Yeah. It's the Ikea effect. It's like you you built it, so it's got much more value to your eyes than than to others. but so d I'm curious, do you do do you do a lot of plot ⁓ a lot of plots? Plots? I think you say plots, right? It is hard for me. ⁓ plots. Do you do a lot of plots in in Jupyter that you do with the agent that the agent is supposed to do? And how does that work? Yeah. What's your experience?
speaker-2: ⁓ it it does a good job. I ha I what I ended up doing was I strip all of the the images when I send the sort of blob to the agent. So by default it'll just like do its best and then give a thing. But then the it has a tool call where it can specifically say like give me back the PNG and then it'll look at it and it can make ⁓ improvements on it. So ⁓ I have softened like I got an obsidian vault so I'm not like completely stuck in two thousand and two. And you can just like rapid fire like plots into an obsidian vault with like some markdown and sort of like DIY notebook. ⁓ and that's really that's really great. the reason why I like great. I every plot I showed you just now was written by Claude. You just have to have you know you have to have you have to have taste. You tell it, you know, I'm French and therefore I need something aesthetic and if you show me something bad, I'm turning you off. And then it gives you what you want.
speaker-0: Because I might I mean so because to me, in my experience, even if I give it like a reference notebook, it will come back with plots which are absolutely a monstrosity and like but stupid stuff like the title will be phone size one hundred and forty eight. And then it will come back and be like, Yep, all done. ⁓ I followed your example, the notebook is ready for you to review. And I'm like No is d it's Absolutely terrible. So th how do you manage to ⁓ I if you manage to make that better, I need to hear about the solution.
speaker-2: Sure, sure. And I mean, part of the answer, like and part of the reason my notebooks were good then and they're good now is because the code's right there and the output's right there. Right. So you really I when I just want to like fire off a bunch of prototypes into a folder, like that's fine. But if I really care about something, I want to be close to the code. And a notebook means that the agent will put the plotting code right there. It'll make the thing, it'll have font size 170, and then instead of typing, I'd like a A dissertation to the thing about how how dare you have put the title so large and need you to do it again. Yeah, just do it. I just go into the code and I say make it eighteen.
speaker-1: Yeah. Yeah. Yeah.
speaker-0: Yeah. So that's where like stuff like if you do that in VS Code for instance, that's the stuff where you would do like the line edit or things like that. Yeah yeah. Okay. So but Tiki but that means you take back the steering wheel. Because that's what I end up doing. Basically I just use the agent for the first version. I know that for most of the code's gonna be fine, let's say eighty percent of the way there. So, you know, like for technicals that stuff. But then for plots If it gets to fifty percent of the way I'm happy. And and then I take the wheel back because just like basically it seems like at least for now, plots with agents is like trying to drive your Tesla on sinuous mountain roads. You know, it's like it starts to do weird stuff that can get you killed and so you're like, ⁓ wait, wait, wait, give me back the steering wheel. ⁓ it is definitely not, you know, like the the one on one where just you do full self driving mode, you're good.
speaker-1: No,
speaker-0: Yeah. Okay. So I thought I'm not completely stupid. I'm not having the worst experience with agents and and plots. That's reassuring for tonight. What about you, Bill? Yeah. Because you told me before the show that actually PTGP, one of the motivations to do that was ⁓ interactions with AI agents. So can you elaborate on that?
speaker-1: No, I Yeah, I mean I mean for for one, I wouldn't be w even doing this. I mean, this has been something that's been kicking around in my head for years, where I'm like, ⁓ if I if someone would just pay me a salary for like two years or longer to just do this thing for fun, that would be awesome. But that's not not gonna happen ever, you know? And so Now that this exists and it's kind of good enough, this is it's it's good enough to build these things. And and I, you know, I guess I do this for work, but I kind of like I kind of hate programming. I hate fixing bugs and I hate like getting stuck in little details that I just want the stupid thing to work because I want to think about whatever problem I'm trying to solve, you know? And it's great for that. So, you know, that it the fact that that stuff exists now is kind of why I could work on this project, but it also kind of got me thinking. It was like, well, if you can just have the robot generate whatever code you want, why do you even need to make a library? Why don't you just make it do the thing you want it to do? And then so I guess it was like, well, unless it can kind of just one shot different GPs and all these back end I mean, it's I think there's still a there's obviously still a place where libraries can you you still need it, of course, but But the fact is that most people are going to be act interacting with it, kind of with an agent in the middle, you know? and so I was thinking, how how should a new library be made in that context? And ⁓ I think I this is what I'm thinking now on and, you know, subject to change, because it's all new, but you know, the the documentation should be, you know, the thing can read all the code. You can just feed it all the code and it can read it and The doc strings are just there to to sort of talk about intent and what it's for. And humans probably aren't going to read it too much. And then but the documentation should be really nice to look at and good for people to read and conscious of people's attention spans because we have much we have much shorter context windows than LLNs do, and there's many other fun things to do that give us dopamine besides reading code all day, at least me. And yeah. Yeah, and it's and so like you know, that's what I was gonna ha so having these skill files, right? Instead of having extensive documentation, I'm like, here's a notebook showing this problem, here's a notebook showing this problem and how to fix it. It can be in skill files. And then when you run across it,
speaker-0: We have it, okay.
speaker-1: the thing, you know, you pointed to the skill files that come with that ship with PTGP and it knows what to do. ⁓ and another thing that is kind of interesting is, you know, when you're using it to write these GP approximations, I mean I'll all they've all been written in other places because I'm not putting anything new in here. So how to cite stuff kind of prominently so that other people's work is actually in there and I'm not just like, you know, vulturing around and cobbling together a bunch of other people's hard work that they had to do real math and think really hard about how to make it. So that's so that's kind of important, I think, for this. ⁓ but but yeah, so it's I think skill files are a really nice way to to sort of debug. And then and then I also think you can kind of interact with the I also think people aren't going to s at least me, I I w I don't sit and just like read documentation, right? Like top to bottom, beginning to end. But you can kind have the LM read it and then it's kind of like choose your own adventure where you can ask a question and then it gives you the explanation and the explanation comes from reading it and and ⁓ you know, you still look at it, but yeah, so I just yeah, how to make a a library that's going to be, you know, has a a purpose before before the LMs can just one shot anything that pops into your head, you know. Yeah. ⁓ whenever that happens, if ever. ⁓ And so it's just like stuff that's tested and it works and stuff that's trickier to just sort of one-shot and you know have have a few kind of like likelihood functions in there, but the stuff that's easy to extend, let the L like have PTG be kind of be ready for people to want to extend it ⁓ to and make it sort of open to that and sort of say where you could extend it and what it has and so I don't know. It's you can tell I'm still still thinking it through. But ⁓ but yeah I think it's interesting though.
speaker-0: I should hope so. It's not like the technology set in stone so
speaker-1: No. Right. Nobody knows, right? Yeah.
speaker-0: Yeah. ⁓ but that's so that's super interesting. You you have a skill file that ships with the package when you peep install it. And look at that. That's like it even Riverside is like super super happy with that.
speaker-1: Exciting.
speaker-0: So you have that in the skill file, so basically that's you going into the skill file and you wrote, Okay, like basically debugging steps where where it's yeah, well, you're gonna work on models that ⁓ that looks like that look like that. Often a problem gonna it's gonna be that. So you're gonna solve it like that way and then iteratively like that, and that's what the skill file is about.
speaker-1: Right. So there's one in there now. Let me see, where was it? But it's ⁓ it's ⁓ for ⁓ the VFE approximation and kind of all the issues that that you can hit just with that one approximation. ⁓ yeah, it's in docs agents PTGP VFE. There's a folder for that. And there's ⁓
speaker-0: Amazing.
speaker-1: There's a skill file for that. And then there's scripts for like making plots to see like preset plots to sort of help you debug stuff. So, you know, here's ⁓ here's a bunch of pitfalls with like links to like each section in the skill file, right? And it'll be like Sigma collapse, Sigma drops to zero, elbow appears to improve, it fits degenerate. Sigma grows during training while elbow plateaus and The condition number of your approximated whatever because of how you chose the induction inducing points is too high. And so there's a cutoff or you have really small eigenvalues there. Because this is all the stuff that happens when you're fitting GP models and you go and I think p I think part of the reason people don't use GPs is they hit one of these things and they go, ⁓ GPs suck. The test error is high. Screw this, I'm back to XGBoost or something, right? And But really there was just like a weird little thing that happened and it's something that you could fix. ⁓ so there's lots of lots of these sort of issues and ⁓ I you know I like to work on I like to work on problems while working on this so that I hit these things and can fix them and keep track of them. and so it's ⁓ yeah, it's like it's a place f where you can kind of collect these things because I think there's a lot of just like there's a lot of folk folk knowledge about how to fix Bayesian models, like when you see, ⁓ the, you know, my my look at my chains and they're in two different places like this. That means I have, you know, multimodal something's going on, but, you know, I should fix the prior because there's an identifiability issue. And ⁓ you know, I think there's a lot of that with GPs out there too, that's you know, maybe more since they're more exp obscure, they're hard it's harder to find that stuff. ⁓ so I want to have that kind of come with it. So that, you know, you don't have to be an expert in this stuff, but it can kind of guide you and so you so you do learn because I you know, I still think you can't run this stuff without knowing what you're doing, right? so it's like Yeah, that's the issue. Exactly. And then you do something really crazy. ⁓ but ⁓ but yeah, so it's it's kind of that in between that
speaker-0: You can run it, that's the problem.
speaker-1: Mm-hmm. Can kinda help people learn and and do it and kinda that balance between, you know, knowing enough to be dangerous and then also having the guardrails kinda built in and kinda help tells you why. So
speaker-0: Yeah. Yeah, I I think that's that's amazing and that's gonna be super helpful to people. So make sure to add all these links to the show notes. ⁓ Jesse, I do want to ask you about, you know, your like the good practices you've seen ⁓ arise from your work and and people's work with interactions with AI agents and how they fit into your work because I do I know you do so many things And you know so many things and so many literatures and so on, so I'm really curious to hear about, you know, what you recommend people to do and what you recommend not to do.
speaker-1: Jupiter notebook.
speaker-2: You should put it in notebooks, that's right. I guess I'll start I'll start with a liter literary reference since you've talked me up. You've heard of this like top like ⁓ Adam Smith had this notion of like a double coincidence, like double miracle. ⁓ it was in the context of barter economies where they say like a barter economy could never work because you need someone who needs exactly a sheep and who has exactly a goat, right? And then we can we can exchange those things, right? Or you're a haircutter and so you're willing to cut hair and you need exactly a pound of wheat, and I happen to be a wheat farmer, so we can do it.
speaker-0: Yeah yeah. So it's basically Catan. Like you you know the the the game.
speaker-2: Yeah, everyone's get to end the microphone. Real life. Exactly, exactly. And you think about think about like developers, like statistical open source developers. Right? Like there's not a double miracle. There's like a hex tuple, like octuple miracle. Because you have to be interested in statistics and like using models to solve a problem. But that's not enough. You have to be interested in the the methodology itself. Like you can't be someone who just wants to like pip install something and run it and get an answer and move on with your life. Like you're You're as interested in the tooling itself as you are as interested in solving some difficult problem. Then in addition to that, you need to be interested in programming per se, because now you're getting into like Git and workflows and collaborating with other people and writing code. And then in addition to that, you need to be a communicator because now you need to write documentation and you have to make sure that you have examples. And then in addition to that, you need to be a public speaker because now you're going to Pi data and you're talking about your stuff and you're promoting yourself and you're putting yourself out there. And like the the the number of people that sit in the middle of that Venn diagram has to be measure zero. And I think one of the powers of LLMs, one of the superpowers of LLMs is it lets you pick your spot and say, I'm someone who's really interested in algorithms. And I'm gonna really focus in on algorithms, and I don't care so much about writing examples. Like Claude, write me an example. First try, no mistakes. And you get a nice example, and you can really dial in on ⁓ focusing on like the algorithms that really motivate you to work. Or you're someone who's really, really interested in studying some specific applied literature, and you come to a Bayesian model and you say, I want to give this a try, and you run it and it doesn't fit, and you get an error message that says, ⁓ your model's not fitting, you should reparameterize it. And you think What are you talking about? Re-parameterize it. I have no earthly idea. And so now you can go to Claude and you can say, Claude, you it says I need to re-parameterize it. What do I do? And it tells you about the difference between a centered and a non-centered parameterization. Or it tells you about the pitfall of having a grand mean and local means and a random effects model. And like, ⁓ silly me, I put a I put a location parameter in all of my random effects. I need to just have one. Right? All of this folk wisdom, right, that kind of Bill is alluding to gets balled up into this big pile of weights. And it lets you say, I want to be the world's foremost expert on schools of fish. And I'm going to use a Bayesian, I'm going to use a GP to estimate the locations of schools of fish at a certain time of year in the Pacific Ocean. And I'm going to be able to do a really, really good job on that because I have access to this side brain that can sort of take on the job of knowing the ins and outs of that piece. My perspective on LLMs fundamentally is that everything is an arbitrage between learning something and doing something now. And if you go like full Gary Tan and you just LLM nonstop, everything is an optimization problem like G stack, let's go, then you've sacrificed every opportunity you have to learn something new. ⁓ on the other hand, if you put on the purity ring and you say like LLM bad, AI is ⁓ taking all the water. then you've completely left on the table the opportunity to build incredible things and to move at pay at really fast speeds in those zones where you've decided for yourself I don't want to be a domain expert in that. But I still want to be able to make progress in my problem. And modern problems require that you have this like octuple miracle of skills and interests in order to make progress on them. So ⁓ I'm I'm super bullish on technology. I love it. I also have felt myself getting dumber. Like I've felt myself making the wrong choices on that split between learning something and doing something because your boss comes in and says, I need this yesterday, like fire it off. No mistakes, first try. And I'm like, well, I guess I'm putting it out with Claude because I do what I have to do. And and I miss those days where I would sit down with a paper and I would get a pencil out and I would rederive the whole thing and really feel like I understood it deeply. And now it's just like, you know, Claude, read this, give me a summary, like let's put it into Pi MC. So there's a trade off there. You have to be cognizant of the trade off, but the same time, like your reach just grows enormously, tremendously, and it's incredible.
speaker-0: Yeah. Yeah. And I feel like so you know what, you should you should come back on the show to talk about that, you know. Like how you use AI agents and and how how that goes into your workflow and the kind of models and so on. And the gotchas. I think the gotchas are the most important because at least in my experience from what I see, I feel like now more and more one of the main skills pun intended. Is is to ⁓ To be able to filter the the LLM's output. And I feel like, you know, LLMs are a bit like YouTube ⁓ the YouTube algorithm where if you wanna learn something, it can sometimes give you some absolutely incredible gems and it's gonna work and that's gonna be awesome. But a lot of the time it's also gonna give you the bullshit, conspiracy theories, videos that look like they are credible and they are not at all. And you actually need to be able to make the to make the difference and that's why to tighten your point, Bill, being an expert in what you're building is much, much easier to do that.
speaker-1: Yeah, no, it's true. I mean, it kinda comes down to some simple stuff at some point too, is they they will just do what you ask them to do, right? And you have to think really carefully about what you're asking it to do and and this thing you're picturing in your head and then what you actually wrote, there's usually a pretty big difference between those things, you know. Yeah. If you were to pass it to someone else and and then you know, of course it's not always gonna do it faithfully either, but ⁓ yeah. It's it's ⁓ it's an interesting problem. It's it's not unlike working with someone else, except you you sort of have to get to know them and what they're good at and what they're not good at and how to work with them effectively and and they're usually, you know, it just like working with people, they can be w far better than you at this thing or that thing, and yeah. Maybe you're a little better at this other thing for a little while and ⁓ and just it's like a how to collaborate with this weird little thing that works very quickly and can make a lot of mistakes quickly and make a big pile of crap very quickly. Or you can do something interesting, right? And you kinda have to not be lazy and say, yes, yes, yes, yes, yes, yes, yes. And which is hard to do sometimes, you know.
speaker-0: No, for sure. For sure. For sure. Especially under time pressure. And like this, I think it's even harder than be thinking a lot about how you ask what you want. Because then you can ask it exactly the way you want it, but then at least two things can happen. It can come back and tell you if d it's done it, but you know, it just thinks it's done it and then it didn't. It literally happened to me several times. and also it can happen that It cannot somewhere in the thinking chain, for some reason cannot answer the question you ask it. But then since it wants to come back with an answer, it will switch what it's gonna answer, but you know, like ever so slightly. And it's gonna be a related question. It can be a good one, but it's gonna be a related question to what you ask, and it's so close to what you asked that it's gonna take you a while to understand that it's not exactly what you asked and and that's like this is very tricky. ⁓ but I think Jesse you wanted to to add something for a while, so I'll I'll give you the floor.
speaker-2: I just wanted to you mentioned the YouTube algorithm and how YouTube has on one hand these incredible learning resources and on the other hand it has just slop. I think there's a more pernicious failure mode that is relevant to LLMs, which is you take something like ThreeBlue One Brown, who's one of my favorite channels on YouTube, does math videos, amazing stuff, and you sit on your sofa in your pajamas with a cup of ice cream. And you watch a video about how to solve an integral with animations, you think, wow, that's incredible. I've learned so much. And then you think, I'm so smart, I'm so great. I watched a video about integrals, and then you close it and you play, you know, whatever. And you have this illusion that you've learned something and that you're like a sophistiqué and everything is delightful, when in reality, you're still a knuckle-dragging crow magnate. And LLMs have the same failure mode where you can convince yourself, like, I made this. You know, like I you shoot off a dashboard and it looks really nice, and you're like, I made a dashboard. And you have to realize that, like, no, no, no, no, no, no, no, no, no, no. You prompted an LLM, you got an interesting response out of it, but you haven't learned anything, you haven't built anything, you got something useful. Right. And you can't you can't lie to yourself and act as though you've improved yourself. by having done that. I think that's that's quite dangerous and it's something that you have to stay on alert for.
speaker-1: Yeah. It's something that's happened and it's like a a thing in data science that's been around a long time is you can write a model, get a fit, boom, done, don't think about it, but then well actually the this feature that comes in, there's something wrong with it. And so your model always predicts this thing when that feature takes a certain value because you didn't look closely enough at things. Right. And you didn't actually learn about this thing that you're modeling, right? Mm-hmm. And I think I think it's nice to think about models as not a tool for doing anything, but like a mechanism for learning about the thing that you're feeding into it, you know. And it it especially with LM's able to just pump stuff out, like you still need to be super critical and care about getting things right. Right. And that's that's kind of difficult to do, you know. 'Cause you can yes, just sit there and feel like, ⁓ look at all the stuff I did and look at all I learned and The thing is the code is running and it doesn't error. It's done, you know? And you know, sometimes you can get great results and and that's the sign that there's a problem you know. And all all this sort of stuff. And I think I don't know if if this it's like this problem's always been around, but L Ms just kind of pour gasoline on it, you know.
speaker-0: Yeah. Yeah, no, exactly. I think it's like d and that's a very good point. I had never thought about that, but yeah, it's a very related problem to what we had before. But now it's like yeah, it gets even worse because it's just like bigger dimension. So Yeah, yeah, yeah. That's yeah.
speaker-1: But they'd be great. I mean, I was working on a icause you sometimes you don't need to know a lot about something too, or and and you need to sort of make a note to maybe I don't know, ask someone who knows or whatever. But like, you know, I was working on an example for PTGP and it was on like ⁓ you know, water main brakes and there's water mains are made out of different pipe materials. And I know absolutely nothing about that, and I don't know anybody who knows anything about that. But LMs know about water main pipe materials, right? And in a pretty straightforward way, like, what is this type of plastic? You know, like I trust the answer that will come out of it for something like that, right? That's pretty like I imagine that's pretty basic for people that know about water mains and how to do water stuff and people that know pipes, right? So it's like it's like basic stuff from other fields. Now you just have access to that, which is really, really nice. And That kind of thing really can help you move on because ⁓ yeah, it's it's it's like I've seen people talk like, Well, models that are really specific and just just trained on this one field are are useful. But also the fact that models are trained on everything is also really, really useful too, because there's there's all these connections between different things and that can be in there. So
speaker-0: Yeah. No, that's true. Yeah, I just think that basically it's like it it lowers the it lowers the the the floor to to entry, the barrier to entry, and like and it also increases the ceiling of what you can achieve, but that also means the standard deviation can get way, way bigger on on what you can get. So ⁓ Yes. ⁓ so I guess we'll talk about that again in the show, like almost every episode now. ⁓ it's in our life. ⁓
speaker-1: Yes.
speaker-0: But I need to to to get you to let you guys go, especially Jesse, it's very late for you, so ⁓ thank you so much for for being on the show guys. I think it was awesome. And Bill, I have to ask you the last questions, of course. So first one, if you had unlimited time and resources, which problem would you try to solve?
speaker-1: ⁓ so I don't know ⁓ what the other answers are, but I'll I'll go with ⁓ I'll take you very literally at unlimited time and unlimited resources and say ⁓ would be really cool to to figure out what the heck's going on in outer space. Because ⁓ hey, we just get to look at it with telescopes and do science like that.
speaker-0: Uh-huh. Yeah.
speaker-1: And it's just little bits of tiny, tiny little bits of photons that come off of a very big place that we we we can't know anything about that. And ⁓ if you had unlimited time and unlimited resources, you could probably learn something about what's going on. So I think that'd be cool. And and world peace and and world hunger.
speaker-0: No, no, no. You just get one and Second question, if you could have dinner with any great scientific mind, dead, alive or fictional, who would it be?
speaker-1: Well, this one's probably a popular answer, but I'm gonna say Richard Feynman. one of the one of the things that got me into science or whatever and switched my major in college when I was a aimless youth, I read ⁓ Feynman's Rainbow, which is a really nice book about that his one of his I think his first year or it he some he was a grad student of his worked with him. ⁓ wrote. I haven't read it in a long time, so I probably got that it's something like that. But it's a really nice book and really inspiring because he's sort of a a person that ⁓ put a lot of passion into a lot of different areas and enjoyed learning about stuff. And ⁓ yeah, so he's sort of a inspirational figure in that way and had a lot of kind of varied interests and ⁓ you know, w life is short. We all have a limited amount of time here and ⁓ you know, you gotta make the best use of it. He's like a Kind of a good, interesting example of of that and kind of appreciating the things that you get to learn about while you're here. So yeah, I don't know, it'd be cool to have dinner with them.
speaker-0: None of that. Yeah. Topical ⁓ answers. Like basically you stayed on the same topic for a
speaker-1: I guess so. Well done. Well done.
speaker-0: Yeah. Impressive. Awesome. ⁓ well guys, I think ⁓ it's time to to call it a show, but ⁓ yeah, ⁓ really awesome to have you here. Jesse, I guess you'll you'll come back on the show very soon. ⁓ we'll talk about all these all this stuff. Be also come back, you know, whenever you have ⁓ you have some new stuff to share. You're always welcome.
speaker-1: Sure. Yeah. I'll start begging, you know, maybe tomorrow again.
speaker-0: Yeah, baking again. Yeah.
speaker-2: This is this is this is Laplace T V, you know, so I'm always happy to
speaker-1: I'm always a reach. ⁓
speaker-0: Where would you go? I mean apparently Bill has hobbies but ⁓ Jesse and I don't, so you know, we're always available.
speaker-1: Mm-hmm.
speaker-0: Awesome. Well guys to to have you on the show. ⁓ listeners take a look at the show notes, they're gonna be great for this one also. And ⁓ well, Bill, JC, thanks again for taking the time and being on the show.
speaker-1: Thank you. Thanks so much.
speaker-0: This has been another episode of Learning Bayesian Statistics. Be sure to rate, review, and follow the show on your favorite podcatcher and visit learnbasetats.com for more resources about today's topics, as well as access to more episodes to help you reach true Bayesian state of mind. That's LearnBaseTats.com. Our theme music is GoodBayan by Beba Brinkman. Fit MC NAS and Mega Rand. Check out his awesome work at Bebabrinkman.com. I'm your host. Alexandora. You can follow me on Twitter at Alex underscore Andora like the country. You can support the show and unlock exclusive benefits by visiting patreon.com slash learnbase tax. Thank you so much for listening and for your support. You're truly a good baby and change your predictions after taking information in. And if you're thinking I'll be less than a maze. Let's adjust those expectations. Let me show you how to be a good daisy. Change calculations after taking fresh data. Those predictions that your brain is making. Let's get them on a solid foundation.
