TRANSCRIPT 🔗
speaker-0: And Adam, actually something you also like, yeah, something I've seen a lot in your work is something that's called the expected information gain. So can you tell us what it is measuring intuitively and why maximizing it is the right objective?
speaker-1: Yeah. First of all, everyone should go and listen to that episode with Desi. I've worked with her pretty closely back at the beginning of her PhD. I'm sure she's got some probably quite complimentary things to say, given her slightly different background. I would really recommend people to check that out. ⁓ Okay, so I to expect an information gain or EIG, as we acronymatize it. So as I was telling you, ⁓ Alex, what... What we do is we have parameters of our experiment and we want to sort of decide them, The questions of your questionnaire or something. And there's a sense that you could choose them optimally. So what would that mean? Well, optimization in this case means that you can associate a score to every one of your designs and then you choose the one with the highest design. And that would be your optimal design. Okay. And then the score is this thing, expected information gain. So it's the amount of information you expect to gain about your model parameters by performing an experiment with a given design. Now, so why don't we unpack that a little bit? Cause I think that's, you know, that's sort of maths language just put out into an English sentence. I'm just trying to think of an example that might, that might kind of ⁓ bring this home, but you, well, If you remember, we said you have uncertainty, right? Your Bayesian model is sort of reflects the uncertainty that you have. If you choose a designer for your experiment, you can also forecast what the datasets would look like through your prior predictive distribution. Right? So now you can take a data set, a prior predictive sample, which is a simulated data set. Then you can do your Bayesian inference, and then you can look at how the uncertainty changed. And then you can repeat that. for various different synthetic data sets. And then you take the average and that's your expected information gain. That's one way to arrive at it. I think what's really cool about it is that, you know, if you actually work through the maths, you can arrive at it in quite different from quite different starting points. So here's a second starting point, which sounds completely different to the first one, but actually arrives at the same quantity. I don't know if you're familiar with this. It's in Brooklyn nine nine. It's this question about you have, I think it's like 12 prisoners on an island. One of them is slightly heavier or lighter. You know, how would you find how would you find out which one it is? ⁓ I don't know if you know how to solve that problem. But basically, the the kind of the way to think about solving that is and that's an experimental design problem, right? Because you have settings, which of these people you want to weigh against which other ones and you have outcomes, which is well, did the scales balance or not balance, right? I don't know if Like, have I told you enough about the setting? Or do want me to just clarify that little problem?
speaker-0: Yeah, maybe clarify it, I think for- For listeners.
speaker-1: Yeah. Okay. I think this is quite a fun, a fun puzzle and people might enjoy it. So I think it's 12. Yeah. I'm pretty sure it's, have, you have 12 people on an island and you were told that one of them is slightly heavier or lighter than all the rest. Okay. And you have access to like a ginormous balance scale, right? One of those old fashioned scales where you weigh two things against the other and it's either level or it's tipping one way or it's tipping the other way. Right. Like my mum has one, is like kind of like a nice old one. ⁓ And you have a stack of weights that you compare to your flower or something. Okay. So now I imagine you have a giant one that can weigh people against each other. ⁓ And then the question is, well, how would you most efficiently find this odd person, right? Of the 12, you've got an odd one out. How would you find them most efficiently with the fewest weighings? That's the puzzle. ⁓ They kind of, I won't tell you the answer. I think it might be fun for people to think about it a little bit. It's in the show and I think they can't solve it in the show. At least if I remember rightly. The best way to think about it is there are three outcomes, right? The scale can either tip left, tip right, or it can balance. And ideally what you would want is for all three of those to be equally represented. Now, why would you want that? It's because you can, you're kind of dividing your search space down as quickly as possible. Right. If you, if you, if all three of them are equally likely with your different hypotheses that you might have. ⁓ then if you see it tipped left, so for example, suppose that it was a third, a third, a third based on the hypotheses that you have, then when, if you see it tipped left, the search space has divided by three. So you've made really good progress on identifying, you know, the true hypothesis. If the probabilities were not. a third, a third, a third, you might have only reduced the search space by quite a small amount. So you can, and you can, you can mathematize this and you can, you can talk about the, right. So the entropy of the outcome is the, right. If you made all three likely. And it turns out that for this experiment, the rate approaches to a maximum outcome. ⁓ so I think hopefully with that hint, people could now figure out this EIG.