บทถอดความฉบับเต็มของ Will machines ever be intelligent?

kernel methods and the   00:02:21.440 --> 00:02:27.600 Bayesian nonparametrics and I started working  more on language models, transformer models,   00:02:27.600 --> 00:02:32.960 with a particular eye towards information theory  and the connection between information theory and   00:02:32.960 --> 00:02:40.080 generative modeling. And that's, kind of, one of  the main things I do today other than, kind of,   00:02:40.080 --> 00:02:45.760 managing the research of people who do much  more interesting work than I do. [LAUGHS] 00:02:45.760 --> 00:02:53.040 BURGER: I have to interject there, Nicolò, because  you dragged a piece of bait across my path. 00:02:53.040 --> 00:02:53.938 FUSI: I figured. 00:02:53.938 --> 00:02:57.280 BURGER: You know, at Microsoft Research,  I have a management rule that I can't tell   00:02:57.280 --> 00:03:00.880 anyone what to do because we hire some of the  best people in the world. You have to trust   00:03:00.880 --> 00:03:07.280 them. And everyone is always completely free to  call BS on me. And so Nicolò was joking there;   00:03:07.280 --> 00:03:13.434 [LAUGHTER] he does not have to toe the party  line. In fact, I encourage him not to. So, so … 00:03:13.434 --> 00:03:16.712 FUSI: I just have to be well-behaved. That's  the only thing I will say. [LAUGHS] 00:03:16.712 --> 00:03:19.200 BURGER: Yeah. Thank you, thank  you for baiting me. [LAUGHS]   00:03:19.200 --> 00:03:22.800 Because he knew exactly what he was  doing. And I love him for it. 00:03:22.800 --> 00:03:25.985 Subutai, can you tell us a  little bit about yourself? 00:03:25.985 --> 00:03:27.760 SUBUTAI AHMAD: Sure. Thank you so much, Doug,   00:03:27.760 --> 00:03:31.760 for having me. I'm really looking forward  to the conversation between us all. 00:03:31.760 --> 00:03:35.600 So I see myself fundamentally as  a computer scientist. You know,   00:03:35.600 --> 00:03:39.760 I've been studying computer science  for longer than I care to admit.   00:03:42.480 --> 00:03:46.880 But something changed for me during my  undergrad years. I decided to minor in   00:03:46.880 --> 00:03:51.760 cognitive psychology, and I started to get  really interested in how the brain works. 00:03:51.760 --> 00:03:56.640 And to me, understanding intelligence and  implementing intelligence was the hardest   00:03:56.640 --> 00:04:02.160 problem a computer scientists could ever solve.  So I got very, very interested in that. You know,   00:04:02.160 --> 00:04:07.760 I couldn't see how to really commercialize that. I  was very interested in making products and stuff.   00:04:07.760 --> 00:04:11.760 So I stopped, you know, working on  that for a while. I did a number of   00:04:11.760 --> 00:04:17.520 startups doing computer vision, you know,  video processing, a lot of that stuff. 00:04:17.520 --> 00:04:22.720 And then when Jeff Hawkins started Numenta  back in 2005 with the idea of really deeply   00:04:22.720 -->

00:04:26.800 understanding how the brain works and  figuring out how to apply that to AI,   00:04:26.800 --> 00:04:32.160 for me, it was like all my worlds coming together.  This, like, this is what I had to do. None of us   00:04:32.160 --> 00:04:36.880 thought [LAUGHS] it would take as long as it  did. We spent the last couple of decades really   00:04:36.880 --> 00:04:41.600 deeply trying to understand neuroscience from a  computer scientist—from a programmer's—standpoint,   00:04:41.600 --> 00:04:45.360 the underlying algorithms. And that's  really what I'm passionate about,   00:04:45.360 --> 00:04:50.960 just trying to translate what we understand  about the neuroscience to today's AI. 00:04:50.960 --> 00:04:54.720 And in terms of what we're working on today,  it's, you know, the human—maybe we'll get into   00:04:54.720 --> 00:04:59.840 some of this—the brain is super efficient in how  it works—power efficient, energy efficient—and   00:04:59.840 --> 00:05:05.480 we're trying to embody those ideas and trying to  make AI a lot more efficient than it is today. 00:05:05.480 --> 00:05:09.440 BURGER: Great. I think we'll get into  efficiency a little bit later in the   00:05:09.440 --> 00:05:13.120 podcast because that's a subject that's  near and dear to my heart, you know,   00:05:13.120 --> 00:05:17.040 being a computer architect  originally by training. 00:05:17.040 --> 00:05:21.680 I want to go back to, you know, one  of the reasons I got involved with   00:05:21.680 --> 00:05:25.360 Numenta is, you know, Subutai and I  have been exchanging emails, like,   00:05:25.360 --> 00:05:30.240 discussing collaborations, you know,  visiting each other through the years,   00:05:30.240 --> 00:05:36.960 and the thing that really stuck with me was  when I read one of the earlier books from Jeff   00:05:36.960 --> 00:05:44.400 on intelligence. And there was an example in the  book that talked about how, you know, the human   00:05:44.400 --> 00:05:49.520 brain learns continuously. I think biological  organisms in general learn continuously. 00:05:49.520 --> 00:05:54.960 And the anecdote that I remember was this anecdote  if you're walking down your basement steps,   00:05:54.960 --> 00:05:58.560 you know, you're walking down the stairs to your  basement and there's one step that’s always been   00:05:58.560 --> 00:06:03.120 a few inches off and you decide to fix it, and  so you raise it so it's even with the others,   00:06:03.120 --> 00:06:07.280 and then the next time you go down the stairs,  you don't remember and you're wildly off and,   00:06:07.280 --> 00:06:11.760 you know, you hit that step, you hit it earlier  or later than you anticipated, you go out of   00:06:11.760 --> 00:06:14.640 balance. You're flailing around. You know, you  get all this adrenaline. You think you're going   00:06:14.640 --> 00:06:19.760 to pitch headfirst down the stairs. Hopefully  you don't. And then the second time you do

it,   00:06:19.760 --> 00:06:24.560 you're a little off balance, but it's not crazy.  And the third time you maybe notice a little bit,   00:06:24.560 --> 00:06:27.680 and the fourth time, it's, like,  it's your basement stairs. 00:06:27.680 --> 00:06:33.280 And so somewhere between that first time  down and the third and fourth times down,   00:06:33.280 --> 00:06:38.080 there are molecular changes in your brain  that have learned the new timing of your   00:06:38.080 --> 00:06:43.360 basement steps. And I remember just that example  vividly from the book. And that got me thinking,   00:06:43.360 --> 00:06:50.400 wow, this is so different from the way  our digital AI works. I'll turn it over   00:06:50.400 --> 00:06:53.680 to you to comment for that and then  I think we'll go into the digital. 00:06:53.680 --> 00:06:58.800 AHMAD: Yeah, no, that's a great example.  I think it's remarkable how our brain is   00:06:58.800 --> 00:07:02.800 constantly modeling our entire  world at such a granular level,   00:07:02.800 --> 00:07:05.760 and we're not even aware of it  perceptually. Like, you know,   00:07:05.760 --> 00:07:11.040 that example of the steps is probably not  … you wouldn't consciously be aware of it,   00:07:11.040 --> 00:07:16.400 yet if something is different about anything in  your world that you're very familiar with, you'll   00:07:16.400 --> 00:07:21.280 instantly notice it. And then you'll, you know,  you'll update your world model, you'll adjust,   00:07:21.280 --> 00:07:26.360 and you'll continue on. It's really remarkable  how the brain’s able to do that so seamlessly. 00:07:26.360 --> 00:07:31.040 BURGER: And a lot of that is based on  neurotransmitters, right? Because there's just   00:07:31.040 --> 00:07:35.600 a … you know, when you have that physical reaction  to “I’m about to pitch down the stairs,” you get   00:07:35.600 --> 00:07:40.000 a flood of transmitters that actually changes the  way your brain's learning or at least the rate. 00:07:40.000 --> 00:07:46.880 AHMAD: Yeah, there's a flood of neurotransmitters  and neuromodulators, as well, that invoke change,   00:07:46.880 --> 00:07:50.800 sometimes very rapidly. Another example,  you know, if you touch a hot stove—that's   00:07:50.800 --> 00:07:57.280 the canonical example—you will learn that very,  very quickly. So there's a lot of chemical changes   00:07:57.280 --> 00:08:03.040 that happen. But it's also really interesting  that we can update things and update our world   00:08:03.040 --> 00:08:07.920 knowledge without impacting everything else  that we know. This is something that's very,   00:08:07.920 --> 00:08:11.920 very different, again, from today's AI  models. We're able to make these changes   00:08:11.920 --> 00:08:15.680 in a very contextual and very,  sort of, fine-grained way. 00:08:15.680 --> 00:08:21.600 BURGER: So, Nicolò, I want to go and talk a  little bit now to transformers. So I think,   00:08:21.600 --> 00:08:25.840 you know, you and I and Subutai  were all working in the AI field,   00:08:26.400 --> 00:08:31.040 you know, many years before 2017,  when the transformer hit. You know,   00:08:31.040 --> 00:08:36.960 I was building, you know, with my team hardware  to accelerate RNNs [recurrent neural networks],   00:08:36.960 --> 00:08:42.800 LSTMs [long short-term memory], you know, which  had this awful loop-carried dependence, you know,   00:08:42.800 --> 00:08:47.840 the bottlenecked computation, and then the  transformer was just much more parallelizable. 00:08:47.840 --> 00:08:54.000 So what do you think's really going on in  these things? And maybe we could start—I know   00:08:54.000 --> 00:08:58.240 you and I have talked a lot about this—maybe  just start with the major blocks. You know,   00:08:58.240 --> 00:09:02.320 you've got the attention layer. You've got  the feedforward layer. You've got, you know,   00:09:02.320 --> 00:09:07.680 the encoder stack and the decoder stack and the  latent space in between. Can you just, kind of,   00:09:07.680 --> 00:09:12.320 walk us through those pieces at a high level  and tell us what you think is going on? 00:09:12.320 --> 00:09:13.680 FUSI: Yeah. Yeah, I mean,   00:09:13.680 --> 00:09:19.548 I have a very opinionated view of  why transformers are so great. 00:09:19.548 --> 00:09:20.111 BURGER: That's why you’re here. [LAUGHS] 00:09:20.111 --> 00:09:22.640 FUSI: Maybe, like, yeah, maybe I’ll  inject it. I don't know. I don't   00:09:22.640 --> 00:09:27.360 think it's a super novel creative  opinion, but it is an opinion. So   00:09:27.360 --> 00:09:31.840 I guess the two principal … the two main  components you already described: the, you know,   00:09:31.840 --> 00:09:36.080 the transformer [read: attention] layers and  the feedforward layers. One way to think about   00:09:36.080 --> 00:09:42.320 them is, how does information in your context  relate to each other and what is every token   00:09:42.320 --> 00:09:46.880 referring to, for instance, in the case  of transformers in language models? 00:09:46.880 --> 00:09:50.400 So by context, we mean, like, the  information you feed through the model,   00:09:50.400 --> 00:09:53.120 that the model keeps continuously  generating and appending to. 00:09:53.120 --> 00:09:54.240 BURGER: So like your chat history. 00:09:54.240 --> 00:09:59.824 FUSI: Your prompt. Your what? Your chat history  or your particular prompt in a chat session. 00:09:59.824 --> 00:10:00.343 BURGER: OK. 00:10:00.343 --> 00:10:04.960 FUSI: That prompt, which is a sequence of words,  gets discretized in a series of tokens. Tokens can   00:10:04.960 --> 00:10:09.920 be individual words, can be multiple words, kind  of, connected together. The way we go from words   00:10:09.920 --> 00:10:16.240 to tokens typically is through an algorithm that  tries to basically collapse as much as possible.  

00:10:16.240 --> 00:10:22.000 Multiple words, like “the dog,” may be just one  token as a first, kind of, level of compression   00:10:22.000 --> 00:10:27.840 to feed into the model. So it just tries to bring  things together as efficiently as possible. 00:10:27.840 --> 00:10:32.640 Then there is, you know, within these models,  there is a transformer layer. This transformer   00:10:32.640 --> 00:10:37.680 layer or this attention layer, sorry,  tries to basically figure out what the   00:10:38.960 --> 00:10:46.480 “the” refers to—the term “the” in “the  dog,” or “the dog jumps on the table,”   00:10:46.480 --> 00:10:51.600 “jumps” refers to the dog. So there is this  kind of, like, mapping that happens. 00:10:51.600 --> 00:10:57.120 And then there is, like, feedforward layers,  which in modern large language models,   00:10:57.120 --> 00:10:59.840 they store a lot of information.  Like, that's kind of, like,   00:10:59.840 --> 00:11:05.840 where the knowledge typically kind of sits in,  the things that the model just knows. You know,   00:11:06.800 --> 00:11:15.360 that, I don't know, if you slam your arm  against [the] cup of water on your table,   00:11:15.360 --> 00:11:19.200 that cup of water falls off the table.  That's something that the model, kind of,   00:11:19.200 --> 00:11:24.800 has baked in through reading a lot about cups  falling off of tables when they’re hit. 00:11:24.800 --> 00:11:31.360 So that's, kind of, those are, for me, the  two fundamental components, and the reason   00:11:31.360 --> 00:11:37.840 why I have an opinionated view is that, you  know, honestly, I do believe that RNNs and,   00:11:37.840 --> 00:11:43.280 you know, even state-space—modern incarnations  of state-space models—are good enough to learn   00:11:43.280 --> 00:11:48.800 over these, you know, language data or  whatever or vision data or audio data. 00:11:48.800 --> 00:11:52.720 The good thing about transformers is that  they do two things very well. One is they   00:11:52.720 --> 00:11:56.240 get out of the way. They don't have this  notion of “everything has to be encoded   00:11:56.240 --> 00:12:00.640 through a state” like recurrent networks.  And two, they do that very computationally   00:12:00.640 --> 00:12:05.040 efficiently as you were saying. There  isn't a computational bottleneck. And   00:12:05.040 --> 00:12:09.520 so they created this nice overhang where  they happen to be the right architecture   00:12:09.520 --> 00:12:13.811 at the right time to unlock enough flow  of information through the model … 00:12:13.811 --> 00:12:14.349 BURGER: Yeah. 00:12:14.349 --> 00:12:16.120 FUSI: … that we could get  through these amazing things. 00:12:16.120 --> 00:12:21.360 BURGER: Let me press you on one thing.  Like, you know, in the attention blocks,   00:12:21.360 --> 00:12:26.160 you can figure out which words or which tokens  relate to which tokens. So I put in the prompt and   00:12:26.160 -->

00:12:33.360 it's finding all the relations and then feeding  those relations up to, you know, the feedforward   00:12:33.360 --> 00:12:39.840 layer—well, the feedforward unit within a layer.  And you said that knowledge is encoded there,   00:12:39.840 --> 00:12:45.120 but then what does it really mean for those maps  to then access knowledge, but then you project it   00:12:45.120 --> 00:12:52.227 back into, you know, the output and then feed it  up to the attention block in the next layer? 00:12:52.227 --> 00:12:52.879 FUSI: Again, yeah. 00:12:52.879 --> 00:12:55.520 BURGER: So it seems kind of weird that  I’d be, like, accessing knowledge and then   00:12:55.520 --> 00:12:59.680 taking that knowledge, merging it, and  going back to another attention map. 00:12:59.680 --> 00:13:04.960 FUSI: Well, you can see it as a mixing  operation that happens in the feedforward   00:13:04.960 --> 00:13:10.320 part of the layer. You know, like, you're  attending, then you're mixing, and, kind of, like,   00:13:10.320 --> 00:13:18.960 reprojecting to some space with higher-information  content or, like, a different level of information   00:13:18.960 --> 00:13:24.560 extraction. And then you're putting it back into,  “OK, so let me do another round of processing”   00:13:25.840 --> 00:13:30.720 and, kind of, attending and then a mix again. And  then I do it again and then I do it again. 00:13:30.720 --> 00:13:34.560 So I think that the information that  is present in the prompt and in the,   00:13:34.560 --> 00:13:39.600 you know, that has been baked into the weights  gather further and further refined. Whether   00:13:39.600 --> 00:13:48.880 that refinement is extraction of structure or  aggregation into higher-level concepts, I'm not   00:13:48.880 --> 00:13:55.040 sure. I think it's just structure gets extracted  and things that are irrelevant get kind of pushed   00:13:55.040 --> 00:13:59.120 away. But that doesn't necessarily mean that it  gets aggregated through the architecture. 00:13:59.120 --> 00:14:04.560 BURGER: So now I'm going to try to, like, restate  what I think I hear you saying. So, you know,   00:14:04.560 --> 00:14:09.920 we're adding information and we're kind of adding  information at a higher level but not necessarily   00:14:09.920 --> 00:14:13.577 throwing away the low-level information,  at least that's not relevant, right? 00:14:13.577 --> 00:14:14.111 FUSI: Yeah. 00:14:14.111 --> 00:14:16.880 BURGER: Because, you know, if the higher-level  stuff depends on the low-level stuff, I have to   00:14:16.880 --> 00:14:21.760 have that first. And so then you get to the top of  the encoder block and you're in the latent space   00:14:21.760 --> 00:14:27.840 with all of that information kind of maximized.  Is that a way to think about it? And if you agree,   00:14:27.840 --> 00:14:32.240 can you talk about what the encoder block  really is and what the latent space is? 00:14:33.440 --> 00:14:40.640 FUSI: I tend to agree, yes. I mean, there  is … you're describing … I think you're   00:14:40.640 --> 00:14:48.800 describing what I think is happening, which is  there is given the context in your prompt and   00:14:48.800 --> 00:14:52.960 given the task that the model perceives  or, like, figures out that you're doing,   00:14:52.960 --> 00:14:59.760 it has to highlight and pull out the relevant  information. And it does that not by summarizing   00:14:59.760 --> 00:15:07.280 layer by layer, but it does it by, you know,  increasing the prominence of that information   00:15:07.280 --> 00:15:13.440 and suppressing other things. So I think that's  ultimately what happens up to the point where   00:15:13.440 --> 00:15:21.440 you reach this beautiful point in concept space,  which identifies both your intent and the things   00:15:21.440 --> 00:15:25.160 in the prompt and in the knowledge of the  model that are necessary to solve it. 00:15:25.160 --> 00:15:28.800 BURGER: And so one last question, and then  I want to go to Subutai for a second. 00:15:29.440 --> 00:15:35.120 So now when we go through the decoder stack, are  we just going the other way and stripping out the   00:15:35.120 --> 00:15:40.880 high-level concepts early and then getting down to  the granular tokens? Or, you know … because you go   00:15:40.880 --> 00:15:45.040 up through the encoder stack, those attention  blocks and feedforward layers, to get to that   00:15:45.040 --> 00:15:49.840 magical latent space. And now we're going to go  the other direction. How do you think about that   00:15:49.840 --> 00:15:55.520 other direction through the decoder stack, which  is the same primitives as the encoder stack? 00:15:56.240 --> 00:16:04.080 FUSI: Same primitives. You can think of it  as kind of the reverse operation. Like you,   00:16:04.080 --> 00:16:09.360 you never lost information throughout. You just  kind of suppress or privileged different kinds of   00:16:09.360 --> 00:16:15.680 information. And now you're basically just  projecting it back out to a space that is,   00:16:15.680 --> 00:16:22.240 you know, intelligible. And it's, kind of,  where the model gets it's … I hesitate to   00:16:22.240 --> 00:16:25.200 use the term reward because it has a  particular implication, but that's,   00:16:25.200 --> 00:16:29.280 kind of, where the loss gets computed and  then gets pushed back through the model. 00:16:29.280 --> 00:16:32.480 BURGER: Right, as you're trying  to evolve and train all those   00:16:32.480 --> 00:16:36.480 parameters—the relationship between words,  the information in the feedforward layers,   00:16:36.480 --> 00:16:41.240 the design of that latent space, and the  extraction of the knowledge from it. 00:16:41.240 --> 00:16:46.480 FUSI: That's right. And so in encoder-decoder  model, you push through the whole thing,   00:16:46.480 --> 00:16:50.720 you decode back to a particular token, 

which for people who don't know, it's,   00:16:50.720 --> 00:16:53.120 like, literally a number out of a vocabulary,   00:16:53.120 --> 00:17:01.642 like word No. 487. And if it was word  No. 1,500, you get, you know, like, … 00:17:01.642 --> 00:17:02.177 BURGER: Something else. 00:17:02.177 --> 00:17:05.840 FUSI: … a bad reward. Yeah. Yeah.  And then … and if you got it right,   00:17:05.840 --> 00:17:09.760 you get a positive signal that then  just flows back through the model. 00:17:09.760 --> 00:17:14.320 BURGER: I'd like to go over to Subutai now. So  after hearing this, you've studied, you know,   00:17:14.320 --> 00:17:19.600 neuroscience and the neocortex and cortical  columns and all of this for a long time,   00:17:19.600 --> 00:17:24.960 and you and I have had lots of debates. Is  the human brain doing something different   00:17:24.960 --> 00:17:29.760 than that? You know, are we  just building latent spaces,   00:17:29.760 --> 00:17:34.240 then extracting? The architecture is very  different, but what's going on under the hood? 00:17:34.240 --> 00:17:37.120 AHMAD: Yeah, the architecture  is very different. You know,   00:17:37.120 --> 00:17:41.200 as Nicolò was describing what happens  throughout a transformer stack,   00:17:41.200 --> 00:17:46.160 I was trying to relay and relate, you know,  what we know in the brain, as well. 00:17:47.040 --> 00:17:52.320 In a typical, you know, transformer model, there  is, at the end of the day, there is a single   00:17:52.320 --> 00:17:58.480 latent space from which the next token is output.  That does not happen in the brain. There are   00:17:58.480 --> 00:18:03.360 thousands and thousands of latent spaces that are,  sort of, collaborating together, if you will. 00:18:04.560 --> 00:18:09.520 You know, a lot of what we publish is under  the moniker the Thousand Brains Theory of   00:18:09.520 --> 00:18:15.280 Intelligence. And Jeff has published a  book a few years ago on that. And that,   00:18:15.280 --> 00:18:19.520 kind of, dates back to discoveries in  neuroscience from the ’60s and ’70s   00:18:19.520 --> 00:18:23.840 by the neuroscientist Vernon Mountcastle,  who was a professor at Johns Hopkins. 00:18:23.840 --> 00:18:24.409 BURGER: Yup. 00:18:24.409 --> 00:18:28.320 AHMAD: And what he discovered … he made  this remarkable discovery that, you know,   00:18:28.320 --> 00:18:32.240 our neocortex, which is the biggest part  of our brain—that's where all intelligent   00:18:32.240 --> 00:18:39.280 function happens—is actually composed of roughly  100,000 what he called cortical columns. 00:18:39.280 --> 00:18:39.880 BURGER: Right. 00:18:39.880 --> 00:18:46.240 AHMAD: And each cortical column is maybe  50,000 neurons. And there's a very complex   00:18:46.240 --> 00:18:50.640 microcircuit and microarchitecture between  the neurons in a cortical column. 00:18:50.640 --> 00:18:56.480 But then there's 100,000 of them,

and  every part of your brain—whether it's   00:18:56.480 --> 00:19:02.080 doing visual processing, auditory  processing, language, thought,   00:19:02.080 --> 00:19:07.760 motor actions—they're all composed of this,  essentially, this same microarchitecture. And   00:19:07.760 --> 00:19:12.720 this was a remarkable discovery. It says  that there's a universal architecture.   00:19:12.720 --> 00:19:17.840 It's not a simple one. It's complex. But  it's repeated throughout the brain. 00:19:17.840 --> 00:19:21.280 And that's where this, you know, the  idea of the Thousand Brains … each of   00:19:21.280 --> 00:19:27.600 these cortical columns is actually a complete  sensory-motor processing system. It has inputs;   00:19:27.600 --> 00:19:32.320 it has outputs. It's getting sensory  input. It's sending outputs to motor   00:19:32.320 --> 00:19:37.920 systems. And it's building, in our  theory, complete world models. So there   00:19:37.920 --> 00:19:41.760 isn't a single latent space. There's  thousands of these latent spaces. 00:19:41.760 --> 00:19:46.800 And each little cortical column is trying  to understand its little bit of the world.   00:19:46.800 --> 00:19:50.480 You know, one cortical column might  be getting, at the lowest level,   00:19:50.480 --> 00:19:55.440 maybe one degree of visual information from  the top right-hand corner of your retina.   00:19:55.440 --> 00:20:00.800 Another one might be focusing on specific  frequencies in the auditory range. You know,   00:20:00.800 --> 00:20:06.080 each one has its own little view of the world,  and it's building its own little world model. 00:20:06.080 --> 00:20:12.240 And then they all collaborate together. There's no  top or bottom here. There's no homunculus in the   00:20:12.240 --> 00:20:19.440 brain. Everything is sort of equal. And they're  all simultaneously collaborating and voting and   00:20:19.440 --> 00:20:26.160 coming up to, you know, what is the, you know,  consistent interpretation of all of these sensory   00:20:27.040 --> 00:20:32.160 inputs that we're getting? What is the  single consistent, you know, concept,   00:20:32.160 --> 00:20:37.440 if you will, and, based on that, make the motor  actions that are most relevant to that. 00:20:37.440 --> 00:20:42.640 So it's a sensory-motor loop. It's a, you  know, it's a constantly recurring system;   00:20:42.640 --> 00:20:48.160 we’re constantly making predictions.  As we discussed earlier, you know,   00:20:48.160 --> 00:20:53.520 we are constantly learning. Every cortical  column is constantly updating its connections,   00:20:53.520 --> 00:20:58.560 constantly updating its weights. It's building  and incrementally improving its world model   00:20:58.560 --> 00:21:05.520 constantly. So it's a massively distributed,  you know, set of processing elements that we   00:21:05.520 --> 00:21:10.240 call cortical columns that are, they're  all equal, operating in parallel. 00:21:10.240 -->

00:21:14.000 So I think there are similarities, for  sure, between them. But at least the   00:21:14.000 --> 00:21:20.000 way I described it, I think it's very  different in its operation than what I   00:21:20.000 --> 00:21:24.880 understand today’s LLMs to be. I don't  know if you agree with that or not. 00:21:24.880 --> 00:21:28.960 FUSI: Yeah, I … To better understand,  I had a question, which is,   00:21:28.960 --> 00:21:32.560 are these cortical columns relying on  the fact that these are essentially   00:21:32.560 --> 00:21:37.440 multiple views of the same process  and those multiple views, like,   00:21:37.440 --> 00:21:41.840 the, you know, the part of the sensory  input that gets allocated or subdivided,   00:21:41.840 --> 00:21:47.760 is it happening at the same time point? So in  other words, if you could artificially delay   00:21:47.760 --> 00:21:53.440 by some time t some cortical columns with respect  to the rest, would the learning suffer? 00:21:53.440 --> 00:21:53.910 AHMAD: Yes, absolutely. Yeah. 00:21:53.910 --> 00:21:57.744 FUSI: And so in other words, how important is  it that it's, kind of, on the same schedule? 00:21:57.744 --> 00:22:01.920 AHMAD: [LAUGHS] Yeah, I mean, that's another … I  mean, LLMs today, you know, you get your input,   00:22:01.920 --> 00:22:05.040 one layer processes it, then the next,  then the next, and the other layers are   00:22:05.040 --> 00:22:09.440 not operating. In the brain, it’s not  like that. Everything is operating in   00:22:09.440 --> 00:22:14.720 parallel asynchronously. And this is important.  They're constantly trying to make predictions   00:22:14.720 --> 00:22:18.640 and so on. So if you were to artificially  slow down some of your cortical columns,   00:22:18.640 --> 00:22:22.000 you would absolutely suffer. Your  thinking would absolutely suffer. 00:22:22.000 --> 00:22:28.560 BURGER: I wanted to interject here just because  this is where … this discussion is where,   00:22:28.560 --> 00:22:33.440 you know, I got super interested in the  difference and then spent a bunch of time   00:22:33.440 --> 00:22:41.120 with Subutai to learn from him. So if I think  about my skin, you know, which is an organ,   00:22:42.160 --> 00:22:48.400 you know, as I understand it, there's a  cortical column attached to each patch of   00:22:48.400 --> 00:22:55.760 my skin and the size of that patch, kind of,  corresponds to the nerve density there. 00:22:55.760 --> 00:22:57.200 AHMAD: That’s right. Yeah. 00:22:57.200 --> 00:23:01.920 BURGER: So in my brain, there is a set of  cortical columns that are skin sensors,   00:23:01.920 --> 00:23:04.800 and I could actually … if I numbered  all the cortical columns in the brain,   00:23:04.800 --> 00:23:08.880 I could draw a map on my skin and say,  “This is No. 72 in this patch. This   00:23:08.880 --> 00:23:14.240 is No. 73 in this patch.” Now are human  cortical columns, like, better than,

say,   00:23:14.240 --> 00:23:19.824 what we see in a mouse? And, of course, this is  a leading question because I know the answer. 00:23:19.824 --> 00:23:24.160 AHMAD: [LAUGHS] Yeah. So, yes, it, you know,  cortical columns in your sensory areas,   00:23:24.160 --> 00:23:29.760 primary sensory areas, each, you know,  pay attention to or get input from a,   00:23:29.760 --> 00:23:33.120 you know, some patch of your skin  somewhere on your body. And there's   00:23:33.120 --> 00:23:37.280 many more cortical columns associated  with your fingertips than, you know,   00:23:37.280 --> 00:23:43.680 a square centimeter of your back, for example.  So there's definitely, you know, areas of sensory   00:23:44.240 --> 00:23:49.120 information that we pay a lot more attention to  and devote a lot more physical resources to. 00:23:49.760 --> 00:23:57.920 In terms of a mouse and humans, it's pretty  remarkable that the cortical columns … so all   00:23:57.920 --> 00:24:01.760 mammals have cortical columns; all mammals  have a neocortex. All mammals have cortical   00:24:01.760 --> 00:24:07.280 columns from a mouse all the way up to humans.  And mice have cortical columns that are very,   00:24:07.280 --> 00:24:13.040 very similar to what a human has. It's not  identical. There are differences. But by and   00:24:13.040 --> 00:24:18.480 large, the architecture of a cortical column  in a mouse is, you know, very, very similar   00:24:18.480 --> 00:24:23.200 to cortical columns in humans. Human cortical  columns are bigger. There are more neurons,   00:24:23.200 --> 00:24:27.432 and there's more detail there, but  essentially, it's the same. And … 00:24:27.432 --> 00:24:29.200 BURGER: Maybe just scaled up a little bit. 00:24:29.200 --> 00:24:34.400 AHMAD: Yeah. So evolution basically discovered  this structure—that it's really excellent   00:24:34.400 --> 00:24:40.560 for processing information and dealing with  it—and then through, you know, very fast in   00:24:40.560 --> 00:24:45.760 evolutionary time, basically figured out that if  you could scale up the number of cortical columns,   00:24:45.760 --> 00:24:53.280 you get more intelligent animals. And that's  what happened very, very fast evolutionarily. 00:24:53.280 --> 00:24:57.280 FUSI: I didn't know about the unevenness  of cortical columns present. Like,   00:24:57.280 --> 00:25:02.880 this is not … I'm not a neuroscientist, and  so this is interesting because one of the   00:25:02.880 --> 00:25:08.640 biggest frustrations with many modern  architectures of models is that they   00:25:08.640 --> 00:25:12.480 deploy a constant amount of computation  no matter what the input is. 00:25:13.120 --> 00:25:21.760 So I go through the same number of layers whether  I'm trying to predict the word “dog” after “the”   00:25:21.760 --> 00:25:26.960 or whether I'm trying to solve, like, give the  final answer to a very complicated math question   00:25:26.960 --> 00:25:33.840 or, you know, whether a theorem was proven or not  in the prompt. And so that's interesting because,   00:25:33.840 --> 00:25:38.800 like, some current instantiations of modern  architecture actually deploy … try to cluster   00:25:38.800 --> 00:25:43.520 things together such that you have a constant  amount of information that you then push   00:25:43.520 --> 00:25:48.640 together through the model. [LAUGHTER] And  so maybe like on my fingertips, I need more   00:25:48.640 --> 00:25:54.000 processing than I need on my elbow because, like,  you know … and so this, kind of, makes sense. 00:25:54.000 --> 00:25:58.560 BURGER: Nicolò is being humble. He was working  on this problem two years ago and told me about   00:25:58.560 --> 00:26:02.880 it. It was one of the things I learned from  you that made me think differently. So … 00:26:02.880 --> 00:26:05.520 FUSI: I just like to refer to people  are working on this ... [LAUGHS] 00:26:05.520 --> 00:26:10.640 BURGER: Random average people who are not  all necessarily brilliant AI scientists. 00:26:11.280 --> 00:26:15.120 So the prediction part of this, though,  is really what's fascinating to me,   00:26:15.120 --> 00:26:20.240 because, again, something else Subutai and I  discussed many years ago, you know, if I'm,   00:26:20.240 --> 00:26:25.920 like, moving my finger towards the table and …  my brain is making predictions because I have   00:26:25.920 --> 00:26:31.680 a world model. It knows a table is there. And the  cortical columns representing that patch of skin,   00:26:31.680 --> 00:26:35.600 as it's getting closer, they're starting  to predict that I'm going to feel something   00:26:35.600 --> 00:26:40.480 that feels like the table. And, yup,  there; I hit it. Prediction met. 00:26:40.480 --> 00:26:46.160 But if I touched it and it felt really  icy cold or super hot or fluffy or not   00:26:46.160 --> 00:26:50.080 there—I pass through it—I'd get  a flurry of activity because the   00:26:50.080 --> 00:26:54.160 prediction wouldn't match the world model,  and that's where learning would happen. 00:26:54.160 --> 00:26:57.760 Subutai, does that sound like the  right model and intuition? 00:26:57.760 --> 00:27:01.920 AHMAD: Yeah, that's definitely a very important  component of it. We're constantly making   00:27:01.920 --> 00:27:06.640 predictions. And as you said, you know,  you're moving your right fingertip down;   00:27:07.760 --> 00:27:12.480 you know, perhaps you've never sat  in this room before or, you know,   00:27:12.480 --> 00:27:16.524 seen this table before, you would still have  a prediction, a very good prediction of it. 00:27:16.524 --> 00:27:17.440 BURGER: Yeah. Because you know what a table is. 00:27:17.440 --> 00:27:22.720 AHMAD: You know what a table is. And if  it was different, you would, you know,   00:27:22.720 --> 00:27:27.680 you would notice it right away. But if your left  hand, which you

weren't paying attention to,   00:27:27.680 --> 00:27:33.600 also felt icy cold, then you would notice  that, as well. So you're actually making not   00:27:33.600 --> 00:27:39.512 just one prediction; you're making thousands and  thousands of predictions constantly about ... 00:27:39.512 --> 00:27:40.416 BURGER: Every cortical column. 00:27:40.416 --> 00:27:43.680 AHMAD: Every cortical column is making  predictions. And if something were anomalous,   00:27:43.680 --> 00:27:47.680 highly anomalous, you would notice  it. So this is something, you know,   00:27:47.680 --> 00:27:52.320 we don't often realize; we're making  very, very granular predictions   00:27:52.320 --> 00:27:56.400 constantly. And when things are  wrong, we do learn from it. 00:27:57.280 --> 00:28:01.600 And the other interesting thing—and this  is, again, possibly different from how   00:28:01.600 --> 00:28:06.800 LLMs work— you know, if I were to tell  you to touch the, you know, the bottom   00:28:08.880 --> 00:28:13.840 surface of the table, you could without, again,  without looking at the table or opening your eyes,   00:28:13.840 --> 00:28:20.240 you would be able to move your finger and touch  the bottom of your table because you have a,   00:28:20.240 --> 00:28:23.654 you know, set of reference  frames that relate to ... 00:28:23.654 --> 00:28:25.280 BURGER: Yup … AHMAD: There you go. Yep. You're able to do it. 00:28:25.280 --> 00:28:27.080 BURGER: I did it! Yeah. Amazing. 00:28:27.080 --> 00:28:29.520 AHMAD: Even though you maybe  never have been in this room;   00:28:29.520 --> 00:28:31.520 maybe you’ve never seen this table  before. It doesn't matter. 00:28:31.520 --> 00:28:34.480 BURGER: I’ve been in this room because we  had to prep for the podcast series. But   00:28:34.480 --> 00:28:36.960 I didn't touch the underside of the  table, that's for sure. [LAUGHS] 00:28:36.960 --> 00:28:41.920 AHMAD: Yeah, exactly. [LAUGHS] So, you know, we  know where things are in relation to each other,   00:28:41.920 --> 00:28:47.360 where our body is in relation to everything,  and we can very, very rapidly learn. And again,   00:28:47.360 --> 00:28:52.680 if the bottom part of the table was anomalous, you  would notice it and potentially remember that. 00:28:52.680 --> 00:28:54.480 FUSI: I'm not going to lie. I was expecting you to   00:28:54.480 --> 00:28:58.360 find something under that table,  [LAUGHTER] like a talk show. 00:28:58.360 --> 00:28:59.360 AHMAD: Or chewing gum or something. 00:28:59.360 --> 00:29:03.920 FUSI: And if you reach under the table, you're  going to find a copy of my paper. [LAUGHS] 00:29:03.920 --> 00:29:07.760 BURGER: [LAUGHS] You know, if I  was smarter and better prepared,   00:29:07.760 --> 00:29:11.520 that's exactly what would have  happened. But, sorry, guys. 00:29:12.880 --> 00:29:18.880 I think you told me something, Subutai, you know,  that … and I'll give a little bit

of preamble. 00:29:18.880 --> 00:29:24.800 So, you know, the brain has these dendritic  networks in each neuron, and they form synapses.   00:29:24.800 --> 00:29:30.400 And so a neuron fires, and that, you know, the  axon of the neuron that's firing will propagate   00:29:30.400 --> 00:29:34.960 a signal through the synapses, which might do a  little signal processing to the dendrites of the   00:29:34.960 --> 00:29:39.840 downstream neurons, and those downstream—the  dendrites can then prime the neuron to fire.   00:29:39.840 --> 00:29:45.040 That's one of the fundamental mechanisms.  And it's the formation of those synapses,   00:29:45.040 --> 00:29:48.480 you know, between the upstream and  downstream neurons, the dendrites,   00:29:48.480 --> 00:29:59.520 that seem to be the basis of learning, and to me,  that feels a little bit like an attention map. 00:29:59.520 --> 00:30:00.312 AHMAD: Yes. 00:30:00.312 --> 00:30:03.520 BURGER: So maybe the dendritic network is  doing something akin to self-attention,   00:30:03.520 --> 00:30:10.640 and we have some work going on in that direction  at MSR. But the thing you told me was that your   00:30:10.640 --> 00:30:18.320 brain is actually forming an incredibly large  number of synapses speculatively. In some sense,   00:30:18.320 --> 00:30:26.160 sampling the world when something happens in case  it will recur. You know, it's a more … maybe it's   00:30:26.160 --> 00:30:30.040 a version of Hebbian learning, right? You know,  things that fire together, wire together. 00:30:30.040 --> 00:30:30.560 AHMAD: Exactly. 00:30:30.560 --> 00:30:33.520 BURGER: But then if that pattern doesn't recur,   00:30:33.520 --> 00:30:38.480 then they get pruned. And I’m just going  to, you know, what is the fraction of your   00:30:38.480 --> 00:30:42.000 synapses to get turned over every three  or four days, you know, ballpark? 00:30:42.000 --> 00:30:45.280 AHMAD: OK. Yeah, I remember this.  This was an absolute mind-blowing   00:30:45.280 --> 00:30:49.760 study in [The Journal of]  Neuroscience. So, you know,   00:30:49.760 --> 00:30:54.960 the way a lot of learning happens in the brain  is by adding and dropping connections. 00:30:54.960 --> 00:31:00.400 In AI models, it's usually strengthening, you  know, high-precision floating-point number,   00:31:00.400 --> 00:31:03.920 making it higher or lower. But you're not adding  and dropping connections. The connections are   00:31:03.920 --> 00:31:10.400 always—in fact, everything is fully connected,  right, between layers. And so in the brain,   00:31:10.400 --> 00:31:15.040 you're always adding and dropping connections.  That's a fundamental mechanism by which we learn,   00:31:17.520 --> 00:31:19.280 one of the fundamental mechanisms. 00:31:19.840 --> 00:31:26.480 What I read in this study is that they  looked at adult mice and adult animals,   00:31:26.480 --> 00:31:32.160 and what they found is that they would look  at the number of synapses that were connected   00:31:32.160 --> 00:31:36.960 over the course of a couple of months—and they  were able to trace individual synapses in this   00:31:36.960 --> 00:31:44.560 particular part of the brain—and what they found  is that every four days, 30% of the synapses   00:31:44.560 --> 00:31:51.840 that were there were no longer there four days  from now. And there was a new 30%. And there's   00:31:51.840 --> 00:31:58.560 a huge number of connections that are constantly  being added and constantly being pruned. And my   00:31:58.560 --> 00:32:03.600 theory of what's going on there is that we're  always speculatively trying to learn things. 00:32:03.600 --> 00:32:08.320 So, you know, there's all sorts of  random coincidences and things that   00:32:08.320 --> 00:32:13.760 we are exposed to on a day-to-day basis.  We're constantly forming connections there   00:32:13.760 --> 00:32:17.920 because we don't know what's actually  going to be required and what's real   00:32:17.920 --> 00:32:21.920 and what's random. Most of it's random;  most of it's not necessary. And the stuff   00:32:21.920 --> 00:32:26.720 that actually is necessary will stay on.  But we're constantly trying to learn. 00:32:26.720 --> 00:32:32.160 This is a part of continuous learning that's often  not appreciated, I think, is that we're constantly   00:32:32.160 --> 00:32:36.640 forming new connections, and then we prune  the stuff that we don't need. In an AI model,   00:32:36.640 --> 00:32:40.720 if you were to do that, it would just go, I  don't know, it would go bananas. [LAUGHTER] 00:32:40.720 --> 00:32:47.273 BURGER: Well, so let's double-click on that.  So when you told me that, the way I … 00:32:47.273 --> 00:32:49.407 AHMAD: This is mind-blowing, this 30%. 00:32:49.407 --> 00:32:49.961 BURGER: It’s crazy. 00:32:49.961 --> 00:32:52.800 AHMAD: Your brain is going to be totally  different a few days from now. 00:32:52.800 --> 00:33:00.000 BURGER: It's so mind-blowing. When you told  me that, I spent some time processing it,   00:33:00.000 --> 00:33:04.000 so a whole bunch of synapses were created  and destroyed during that time. 00:33:04.000 --> 00:33:08.960 But it just made me think that we have,  you know, we have all of these columns   00:33:08.960 --> 00:33:16.800 getting all of this input continuously. You  know, eyes, hearing, smell, taste, skin,   00:33:16.800 --> 00:33:22.880 heat, and then, you know, interactions with  people, and then planning and experiences,   00:33:22.880 --> 00:33:30.160 just at every level. And they're constantly  sampling all this noise coming in and basically   00:33:30.160 --> 00:33:36.240 filtering out the noise. It's like, kind of,  like a low-pass filter. But when something   00:33:36.240 --> 00:33:42.040 statistically significant recurs, it's going  to lock and then become persistent. 00:33:42.040 --> 00:33:46.800 AHMAD:

Yeah, yeah, I think so. There's so much  that's happening, and you’re constantly learning,   00:33:46.800 --> 00:33:51.120 and, you know, when you touch a hot stove  or something, there's a flood of dopamine   00:33:51.120 --> 00:33:57.200 specific to those areas that caused these  synapses to strengthen very, very quickly.   00:33:58.720 --> 00:34:04.297 You know, most of these synapses that are  learned are very, very weak synapses. 00:34:04.297 --> 00:34:04.821 BURGER: Yup. 00:34:04.821 --> 00:34:07.120 AHMAD: And so, yeah, you know,  when you look … in this study,   00:34:07.120 --> 00:34:12.160 they also quantified the turnover in, kind  of, strong synapses versus weak synapses.   00:34:12.160 --> 00:34:16.800 And it's comforting to know that the strong  synapses stay there. It's really these weak   00:34:16.800 --> 00:34:20.440 synapses that are constantly added and dropped.  And then some of them will become strong. 00:34:20.440 --> 00:34:25.280 BURGER: Now I want to go back … return  to Nicolò, but with an observation. 00:34:25.280 --> 00:34:32.560 So when I'm training a transformer, it's  also a prediction-based system. You know,   00:34:32.560 --> 00:34:39.280 I'm running … I have my input in the training  set; I have my masked token or the next token   00:34:39.280 --> 00:34:45.920 I'm trying to predict. I run it through. I look  at how successfully did it make that prediction,   00:34:45.920 --> 00:34:51.040 and the worse it was, the, sort of,  the steeper the error, you know,   00:34:51.040 --> 00:34:56.720 I drive back through the network. So, you  know, if it's spot-on, I don't learn very   00:34:56.720 --> 00:35:00.960 much. But if the prediction is way off,  I've got to change a bunch of stuff. That   00:35:00.960 --> 00:35:05.680 sounds analogous to what Subutai was just  describing with the cortical columns. 00:35:05.680 --> 00:35:14.720 FUSI: No, that's right. I mean, with,  I don't know, with one big pet peeve of   00:35:14.720 --> 00:35:18.219 mine in pretraining, in particular around  pretraining these language models. 00:35:18.219 --> 00:35:18.746 BURGER: OK. 00:35:18.746 --> 00:35:21.520 FUSI: So again, for context, like,  language models in particular, but,   00:35:21.520 --> 00:35:27.040 you know, many other instantiations of large  models, are trained in a few phases usually.   00:35:27.040 --> 00:35:33.120 One of them is pretraining, where you have  some ground truth text and you remove,   00:35:33.120 --> 00:35:37.360 let's say, just the last word, and then you  ask the model to predict the last word. And   00:35:37.360 --> 00:35:42.000 that's when you get that loss. Do you get the  word right? Do you get the word wrong? 00:35:43.680 --> 00:35:46.080 One of the big problems that I have is that,   00:35:47.040 --> 00:35:52.400 you know, in human experience, we do not  get feedback every single thought. 00:35:52.400 --> 00:35:56.000 The problem with language models, the way we  are training them, at least in pretraining,   00:35:56.000 --> 00:36:00.640 is that they do a thing called teacher forcing. So  they guess the word, then they get immediately the   00:36:00.640 --> 00:36:04.480 signal, and then the right word gets filled  in, and then they predict the next one. 00:36:04.480 --> 00:36:09.360 So when you go through, like, a passage  of text, you constantly get this reward.   00:36:09.360 --> 00:36:13.360 And it's such a bizarre way to train a  model. It's necessary because you want   00:36:13.360 --> 00:36:20.320 a lot of flow of supervision. Like, you want,  like, a lot of supervision to essentially use   00:36:20.320 --> 00:36:25.040 all the computation available. But at the same  time, it actually makes the models arguably a   00:36:25.040 --> 00:36:29.840 little bit worse than what they would be if you  had enough compute to train them without this. 00:36:29.840 --> 00:36:32.444 I went on a tangent just because  it's a pet peeve. [LAUGHS] 00:36:32.444 --> 00:36:35.600 BURGER: It's a really important point,  though, because your goal when you're   00:36:35.600 --> 00:36:42.480 training a model is to get to your loss  target with the minimal cost and time.   00:36:42.480 --> 00:36:46.400 Or, of course, like, fixed budget  and, like, lowest loss target. 00:36:46.400 --> 00:36:51.280 But, you know, biological systems, also, their  goal is survival with energy minimization. And so,   00:36:51.280 --> 00:36:55.200 like, once you've built a world model that  works, right, like touching the table,   00:36:55.200 --> 00:36:58.560 touching the underside of the table—nope,  still nothing exciting there—like,   00:36:59.120 --> 00:37:04.400 it takes very little energy to do that. And  I think a tragedy is that we all have these   00:37:04.400 --> 00:37:09.040 supercomputers in our heads. You know, the  neocortex is what, about 10 watts? And it's   00:37:09.040 --> 00:37:14.400 this amazing thing, right, that can compose  symphonies. But once we have a world model,   00:37:14.400 --> 00:37:19.360 a lot of us just stop learning because it's  comfortable, right. You don't have to perturb   00:37:19.360 --> 00:37:24.400 the state. You can go through … and, you know,  I mean, how many of us go through every day and   00:37:24.400 --> 00:37:28.480 all of our predictions succeed [LAUGHTER],  and there's no surprises, you know? 00:37:28.480 --> 00:37:32.400 So all the new synapses get swept away,  right. That's not a goal of pretraining   00:37:32.400 --> 00:37:35.600 because then you're just wasting energy. But  we're trying to minimize energy consumption.   00:37:36.240 --> 00:37:40.080 So it does feel, kind of,  aligned to me in some sense. 00:37:40.080 --> 00:37:44.000 So I've got a straw man I want to  hit you with, but before we do,   00:37:45.440 --> 00:37:51.280 Nicolò, I want you to talk about your view  on compression, like LLMs as

compressors,   00:37:51.280 --> 00:37:53.600 because I know this is something  you're very passionate about and   00:37:53.600 --> 00:37:56.400 opinionated about. And I've learned  a lot from you on this, too. 00:37:57.360 --> 00:38:02.240 And then, Subutai, after this, I'd like  to hear your biological response. I mean,   00:38:02.240 --> 00:38:07.120 your response from a biological  perspective. [LAUGHTER] And … 00:38:07.120 --> 00:38:08.360 AHMAD: You'll get both. 00:38:08.360 --> 00:38:11.200 BURGER: That's right, of course.  And then I want to try … I want   00:38:11.200 --> 00:38:15.600 to throw out this hybrid straw man. So,  Nicolò, tell us about compression. 00:38:15.600 --> 00:38:22.400 FUSI: The view is that basically the  generative models are compressors in   00:38:22.400 --> 00:38:27.440 an information theoretic sense, and so  trying to come up with a better generative   00:38:27.440 --> 00:38:32.388 model is equivalent to trying to find the  best compressor for some data. And … 00:38:32.388 --> 00:38:35.120 BURGER: Now when you say compressor,  do you mean lossless or lossy? 00:38:35.120 --> 00:38:37.120 FUSI: I mean lossless. 00:38:37.120 --> 00:38:37.840 BURGER: OK. 00:38:37.840 --> 00:38:45.760 FUSI: You can basically look at literally my  much-maligned objective function that you use   00:38:45.760 --> 00:38:52.800 for pretraining, which is, you know, next-token  prediction, and you can basically draw a complete   00:38:53.440 --> 00:38:56.880 parallel to what you would do if  you were trying to come up with the,   00:38:56.880 --> 00:39:00.880 you know, try to do compression, which is  coming up with the shortest possible code   00:39:02.240 --> 00:39:03.920 for something that you're trying to compress. 00:39:03.920 --> 00:39:10.080 And so the two things are the same, and it,  kind of, fits into a broader picture that,   00:39:10.080 --> 00:39:16.720 you know, like, goes back to Occam's razor  and Kolmogorov complexity and Solomonoff's   00:39:16.720 --> 00:39:23.520 principle of induction, which is, you want  short descriptions for likely things that   00:39:23.520 --> 00:39:27.440 happen in the world and you want your  algorithm that produces those short   00:39:27.440 --> 00:39:30.640 descriptions to be also short. That's the  minimum description length principle. 00:39:30.640 --> 00:39:36.000 And I do feel like it fits in, kind  of, also what you were saying about   00:39:36.000 --> 00:39:41.280 the concept of you have a good world  model, why look for surprise? Because   00:39:41.280 --> 00:39:46.640 it simultaneously affects both terms, both  the algorithm, like your own world model,   00:39:46.640 --> 00:39:50.080 but also the loss that you incur  when something unexpected happens. 00:39:50.080 --> 00:39:55.120 And so if I'm an agent in the  world trying to minimize the   00:39:55.120 --> 00:39:59.520 minimum description length of the  world,

I’d like to go and seek some   00:39:59.520 --> 00:40:04.160 in-distribution data such that I don't  bump up my surprise term too much. 00:40:04.160 --> 00:40:07.840 BURGER: Right. And I think you  said at some point that, you know,   00:40:07.840 --> 00:40:11.440 when I'm training a model, even though  you took the same loss point, you know,   00:40:11.440 --> 00:40:17.760 between Model A and Model B, if I have a  steeper loss curve in Model A than Model B,   00:40:17.760 --> 00:40:24.160 you know, it's getting to a better, sort  of, compressed-based vocabulary faster,   00:40:24.160 --> 00:40:28.360 which makes it more general. The shape of that  curve matters from a compression perspective. 00:40:28.360 --> 00:40:32.160 FUSI: Yeah. I mean, I think  it would help here to expand   00:40:32.160 --> 00:40:33.672 on what I was talking about in terms of, … 00:40:33.672 --> 00:40:34.262 BURGER: Yes. Please. 00:40:34.262 --> 00:40:36.560 FUSI: … like, minimum description length  principle. The minimum description length   00:40:36.560 --> 00:40:40.320 principle is basically the loss  of the model you're training;   00:40:40.320 --> 00:40:46.640 that's one component. And so it's a sum over the  mistakes you make at predicting or, you know,   00:40:46.640 --> 00:40:52.560 the mistakes you make at predicting each word.  And that's one term. And the other term is how   00:40:52.560 --> 00:40:57.812 long it takes you in code to describe the  model and the training procedure, … 00:40:57.812 --> 00:40:58.348 BURGER: Right. 00:40:58.348 --> 00:41:01.342 FUSI: … to get to that training curve,  to produce that training curve. 00:41:01.342 --> 00:41:01.884 BURGER: Right. 00:41:01.884 --> 00:41:06.480 FUSI: So, yes, if you look at collectively,  one term is, kind of, fixed. It's an amount   00:41:06.480 --> 00:41:10.720 of code it would take you to write out a  language model, for instance, in code. Like,   00:41:10.720 --> 00:41:17.520 literally implement it, not the weights, just  implement the initialization of it and then the   00:41:17.520 --> 00:41:21.680 training loop. And then on the other side, you  have this training loss that gets generated as   00:41:21.680 --> 00:41:26.960 you start observing data. And, of course, because  it's a sum, you want to minimize really the area,   00:41:27.600 --> 00:41:34.640 like, you want to minimize the sum. And so,  like, a flatter curve is much better than,   00:41:34.640 --> 00:41:39.240 like, the steeper curve, you know, even if it  ends up at the end to be slightly better. 00:41:39.240 --> 00:41:42.240 BURGER: Yeah. Concave is better than convex. 00:41:42.240 --> 00:41:45.032 FUSI: Among other things, yes. [LAUGHTER] 00:41:45.032 --> 00:41:52.880 BURGER: Sorry. So, you know, I think that we  could do a whole episode on this compression   00:41:52.880 --> 00:41:58.240 view because it's really fascinating. And  the lossless part of it is what

blew my mind.   00:41:58.960 --> 00:42:03.200 And I think, you know, I'm guessing there  are multiple camps here, and you're squarely   00:42:03.200 --> 00:42:08.400 in one camp, so I'm guessing we'll get a  bunch of feedback from the other camps. 00:42:08.400 --> 00:42:14.360 So, Subutai, you know, can I think of  cortical columns as compressors? 00:42:14.360 --> 00:42:22.640 AHMAD: Yeah, it's a good question. You know,  I, you know, there's so much in the compression   00:42:22.640 --> 00:42:27.440 literature that you can draw insight from.  You know, if you look at the representations   00:42:27.440 --> 00:42:35.120 in cortical columns and that populations that  neurons have, you know, some of the things you   00:42:35.120 --> 00:42:41.040 have to deal with are that the brain doesn't have  a huge nuclear power plant attached to it. 00:42:41.040 --> 00:42:47.680 You know, we only have 12 watts or so to process  everything we want to do, and the representations   00:42:48.240 --> 00:42:54.000 that evolution has discovered are incredibly  sparse. And what that means is that you may   00:42:54.000 --> 00:42:59.600 have thousands and thousands of neurons in a  layer, but only about 1% of them will actually   00:42:59.600 --> 00:43:06.960 be active at a time. And so it's a very small  subset of neurons that are actually active. 00:43:07.920 --> 00:43:13.200 I don't know about this minimum description  length, whether that applies. I can say a   00:43:13.200 --> 00:43:17.920 couple of things about that. There's, you know,  by and large, the representations are very sparse   00:43:17.920 --> 00:43:23.737 when you're predicting well. When you see a  surprise, there's a burst of activity. 00:43:23.737 --> 00:43:24.264 BURGER: Yup. 00:43:24.264 --> 00:43:27.912 AHMAD: When there's something that's unusual,  there's a lot more neurons that fire, and ... 00:43:27.912 --> 00:43:29.320 BURGER: That's why learning is tiring! 00:43:29.320 --> 00:43:33.280 AHMAD: That's why learning [LAUGHTER] …  exactly. No, no, that's right, that's right. 00:43:33.920 --> 00:43:37.120 And so what we think is  happening is that, you know,   00:43:37.120 --> 00:43:41.280 the actual representation of something is a very  small number of neurons. When you're surprised,   00:43:41.280 --> 00:43:44.480 there may be many things that are  consistent with that surprise,   00:43:44.480 --> 00:43:49.200 and so your brain represents a union  of all of those things at once. 00:43:49.200 --> 00:43:51.520 And when you have a very sparse representation,   00:43:51.520 --> 00:43:55.760 you can actually have a union of many, many  different things without getting confused. So   00:43:55.760 --> 00:44:01.520 that's what we think is going on there. So it is  a very compressed, very efficient representation.   00:44:02.480 --> 00:44:06.720 And because it's such a small percentage  of neurons that are firing, we are very,   00:44:06.720 --> 00:44:13.680 very parsimonious in how we represent things  and extremely energy efficient metabolically. 00:44:13.680 --> 00:44:18.800 BURGER: I wanted to get to the  efficiency point, but before I do,   00:44:18.800 --> 00:44:24.000 you know, you talk about this 1, you know, 1 to  2% of the neurons firing. But it's, actually,   00:44:24.000 --> 00:44:27.331 the brain is actually much sparser  than that at a fine grain, right? 00:44:27.331 --> 00:44:27.881 AHMAD: Yes, yes. 00:44:27.881 --> 00:44:30.320 BURGER: Because, you know, you  have 1% of the neurons firing,   00:44:30.320 --> 00:44:34.160 but they aren't connected to all  the other neurons in the region. 00:44:34.160 --> 00:44:35.280 AHMAD: That's right. Yeah. 00:44:35.280 --> 00:44:39.040 BURGER: So really the sparsity should  be the product of the connectivity   00:44:39.040 --> 00:44:41.840 fraction times the activity factor. 00:44:41.840 --> 00:44:43.040 AHMAD: Yeah. Yeah. 00:44:43.040 --> 00:44:46.560 BURGER: Right. That's about one out  of 10,000. Something like that. 00:44:46.560 --> 00:44:52.160 AHMAD: Exactly. Yeah. So something like maybe 1%  of the neurons are firing at any point in time,   00:44:52.160 --> 00:44:56.320 and maybe 1% of the connections that are possible   00:44:56.320 --> 00:45:02.480 are actually there at any point in  time. So it's a very, very small,   00:45:02.480 --> 00:45:06.560 you know, subnetwork through this massive  network that's actually being activated,   00:45:06.560 --> 00:45:11.360 a tiny percentage of neurons going through a  very, very tiny piece of the full network. 00:45:12.160 --> 00:45:14.080 You know, it's common to,  you know, some people say,   00:45:14.080 --> 00:45:19.440 “Oh, we're only using 1% of our brain.” That's  not true. It just means at any point in time,   00:45:19.440 --> 00:45:24.880 you're only using 1%, but at other points in  time, a different 1% is being used. So, you know,   00:45:24.880 --> 00:45:30.280 the activity does move around quite a bit. But,  any point in time, it's extremely small. 00:45:30.280 --> 00:45:33.200 BURGER: So, OK, the sparsity, I think, you know,   00:45:33.200 --> 00:45:38.720 the representation—how the brain is doing this  compression biologically—is super fascinating.   00:45:38.720 --> 00:45:43.440 And I want to go on a little bit of a  detour now to efficiency. So I remember   00:45:43.440 --> 00:45:50.640 in 2017 when in MSR we were building, you  know, hardware acceleration for RNNs. 00:45:50.640 --> 00:45:54.960 And then the transformer hit, and they were  optimized, you know, to be highly parallelizable   00:45:54.960 --> 00:46:02.640 across this quadratic attention map for GPUs. The  way I would describe it is that that transition   00:46:04.080 --> 00:46:08.880 to semi-supervised training moved us from  an era when we were really data limited,   00:46:08.880 --> 00:46:15.360 like you had to have good

high-quality  labeled data, to you were compute limited. 00:46:15.360 --> 00:46:19.920 And when that transition happened, we  hockey-sticked from, “I'm building faster   00:46:19.920 --> 00:46:26.400 machines but I'm limited by data” to the bigger  machine I can build, as long as I have enough,   00:46:26.400 --> 00:46:31.200 you know, unlabeled data of high quality, the  better I can do with the model. And so we went   00:46:31.200 --> 00:46:38.000 on the supercomputing arms race, and now we're  building these, like, just gargantuan machines. 00:46:41.040 --> 00:46:46.320 And really, we've kind of been brute-forcing it.  I mean, we've done a lot of things to optimize,   00:46:46.320 --> 00:46:51.440 like quantization, you know, and other  and, you know, a better process node,   00:46:51.440 --> 00:46:55.120 you know, a better, more efficient  tensor unit design. But to first order,   00:46:55.120 --> 00:46:59.520 we've been training bigger models  by building bigger systems. 00:46:59.520 --> 00:47:08.880 And I just wonder, do you think that the brain  at this 10 to 12 watts in the neocortex just   00:47:08.880 --> 00:47:14.400 has a fundamentally more efficient learning  mechanism? Or do we think that, you know,   00:47:14.400 --> 00:47:18.880 what we're doing in transformers in the  most advanced silicon is as efficient,   00:47:18.880 --> 00:47:21.440 we're just building much  larger, more capable models? 00:47:21.440 --> 00:47:25.920 AHMAD: Oh, I think without a doubt,  transformers are extremely inefficient   00:47:25.920 --> 00:47:31.200 and very, very brute force. We touched on this  a little bit earlier in the attention mechanism,   00:47:31.200 --> 00:47:35.680 where we’re, you know, transformers  are essentially comparing every token   00:47:35.680 --> 00:47:39.840 to every other token. I mean, there are  architectures which reduce that, for sure,   00:47:39.840 --> 00:47:44.960 but it's essentially an n-squared operation.  And we're doing this at every layer. 00:47:44.960 --> 00:47:50.240 I mean, there's nothing like that in  the brain. Our processing, you know,   00:47:50.240 --> 00:47:56.080 in some sense, the context for the very next  word I'm about to say is my entire life,   00:47:56.080 --> 00:48:01.200 right? And the amount of time I take  to take the next word doesn't depend   00:48:01.200 --> 00:48:06.400 on the length of the context at all. It's  a constant time dependence on context. 00:48:06.400 --> 00:48:13.600 So it's a significant, you know, reduction  in the compute that's required. You can kind   00:48:13.600 --> 00:48:18.480 of think about, like the brain—I think  has somewhere around maybe 70 trillion   00:48:18.480 --> 00:48:23.360 synapses. When I say the brain, I mean the  neocortex, has about 70 trillion synapses.   00:48:23.360 --> 00:48:28.560 And it's using only 12 watts. And a synapse  is roughly equivalent to a parameter. 00:48:28.560 --> 00:48:35.520 And

if you were to take the most efficient GPUs  today and try to run a 70 trillion parameter   00:48:35.520 --> 00:48:41.840 model, it would be something like a megawatt  of power. It's tens of thous ... it's orders   00:48:41.840 --> 00:48:49.200 of magnitude more inefficient than what our  brain is doing. So I absolutely believe that. 00:48:49.200 --> 00:48:51.840 BURGER: The metric I use, to go  back to your point, you know,   00:48:51.840 --> 00:48:56.080 is, this is something, I think we talked  about this back in the day, right? When,   00:48:57.520 --> 00:49:00.800 you know, after this kicked off for a few  years, we were trying to project, like,   00:49:00.800 --> 00:49:04.800 how far would this go under the current model  to inform the research and the directions you   00:49:04.800 --> 00:49:07.600 took. Which is why I got so interested  in sparsity and working with you. 00:49:07.600 --> 00:49:11.280 And we would look at a training run and just  say, how many joules did it take to train the   00:49:11.280 --> 00:49:17.840 whole model? How many parameters do we have? And  sort of what's our parameters per joule? And,   00:49:17.840 --> 00:49:22.480 if by that metric, you know, we were off by  many orders of magnitude where the brain is,   00:49:22.480 --> 00:49:26.720 but I don't know that that's the right  metric. So any thoughts on that? 00:49:26.720 --> 00:49:31.120 AHMAD: Yeah. I mean, in some  ways, you know, transformers,   00:49:31.120 --> 00:49:35.894 you know, embody more knowledge  in them than any human has. 00:49:35.894 --> 00:49:36.425 BURGER: Right. 00:49:36.425 --> 00:49:41.112 AHMAD: It has memorized, you know, the entire  internet's worth of knowledge, essentially. 00:49:41.112 --> 00:49:42.873 BURGER: All scientific papers ... 00:49:42.873 --> 00:49:45.920 AHMAD: All scientific papers.  You know, good and bad, whatever,   00:49:45.920 --> 00:49:48.800 you know, it has memorized everything.  So that's something that, you know,   00:49:48.800 --> 00:49:54.800 humans just cannot do. So there's definitely stuff  that's better in transformers than humans. 00:49:54.800 --> 00:50:00.000 But fundamentally, I think, you know,  we're extremely efficient in how we process   00:50:00.000 --> 00:50:05.600 the next token or the next bit of information  that's coming in. And I think there's a lot   00:50:05.600 --> 00:50:10.880 we can learn from the brain and apply  to LLMs and future AI models there. 00:50:10.880 --> 00:50:14.160 FUSI: I was going to ask a question related  to that because ... forget memorizing the   00:50:14.160 --> 00:50:19.040 internet. But let me give you another example that  transformers do really well. And I'm wondering,   00:50:19.040 --> 00:50:25.440 like, you know, the human aspect of this or the  brain aspect of this because transformers, because   00:50:25.440 --> 00:50:28.640 of the n-square computation, they're really  good at stuff, like a needle in the haystack. 00:50:28.640 --> 00:50:32.960 So I can tell you right now, I can speak, I can  talk to you, and I can tell you the password is   00:50:32.960 --> 00:50:37.600 something silly like “podcast microphone  blue,” whatever. That's the password. And   00:50:37.600 --> 00:50:43.200 then I can proceed and read the entire Odyssey  or a bunch of other books to you out loud for   00:50:43.200 --> 00:50:48.480 the next 5 or 6 hours. And then I can ask  the transformer, what was the password? And   00:50:48.480 --> 00:50:54.480 transformer will do this nice n-square computation  many times, and it will spit out the password. 00:50:54.480 --> 00:50:59.040 A human, you know, there will be a decay  of that password. And then at some point,   00:50:59.040 --> 00:51:03.520 it won't remember, and depending on the human,  it may be in the first chapter of the Odyssey or   00:51:03.520 --> 00:51:08.960 like at the end, but … so fundamentally the type  of computation that is done is very different. So   00:51:10.000 --> 00:51:12.640 it always makes me wonder about the  efficiency because it's just, like,   00:51:12.640 --> 00:51:17.760 it's a different type of computation. So the  efficiency of … like, efficiency is kind of like,   00:51:17.760 --> 00:51:24.000 what are you doing divided by how good are  you at doing it. And so when the things we're   00:51:24.000 --> 00:51:28.320 doing are so incomparable in many ways,  that always makes me ... always troubles   00:51:28.320 --> 00:51:34.073 me a little bit. I don't know... I don't know  if there's any question in there. [LAUGHTER] 00:51:34.073 --> 00:51:37.680 AHMAD: Yeah. I mean, transformers can  do the stuff that humans find very,   00:51:37.680 --> 00:51:43.520 very difficult to do. Absolutely. You know,  maybe there's a way to get the best of both.   00:51:43.520 --> 00:51:49.520 I don't know. You know, I don't know  that it's fundamentally necessary to   00:51:49.520 --> 00:51:54.707 have such brute-force computation  to get all of these features. 00:51:54.707 --> 00:51:55.402 FUSI: That's right. 00:51:55.402 --> 00:51:57.680 BURGER: Yeah. Yeah, it is a  weird thing because, you know,   00:51:57.680 --> 00:52:01.200 this is why memory palaces work so  well. Like, there is a way, though,   00:52:01.200 --> 00:52:05.840 for a human to remember that my microphone  is gray. It's not actually blue, Nicolò. 00:52:05.840 --> 00:52:09.032 FUSI: Mine is blue. You don't see it. It's  off camera. You see, your world model … 00:52:09.032 --> 00:52:11.600 BURGER: It’s off camera. Yeah, I  know. I was just teasing you. 00:52:11.600 --> 00:52:14.800 But there's a way, like, if I can  just connect it to enough things,   00:52:14.800 --> 00:52:19.120 get that connectivity graph, then I'll  remember it because it's captured the   00:52:19.120 --> 00:52:23.040 signal out of the noise and connected  to enough things I can retrieve it.   00:52:23.680

--> 00:52:27.360 And retrieval would be a whole other topic  we don't have time to get into today. 00:52:27.360 --> 00:52:32.560 But I do … now, I want to go to the straw  man. So let's take continual learning off   00:52:32.560 --> 00:52:39.200 the table. Let's imagine that, as I go  through my day, I'm just saving all of   00:52:39.200 --> 00:52:46.640 the sensory data to put in my training set.  And now imagine that I take 100,000 little   00:52:46.640 --> 00:52:52.160 transformer blocks, and I'm training  them each with what they're seeing. 00:52:52.960 --> 00:52:57.360 OK, I replay the day so I don't have to, again,  I don't have to worry about continuous learning   00:52:57.360 --> 00:53:03.600 and whatever cross-cortical column, you  know, routing feature of the outputs,   00:53:03.600 --> 00:53:07.200 the inputs, and there's—Subutai, we've  talked about this—there’s a complex set   00:53:07.200 --> 00:53:13.600 of wiring there to bring features from here to  there that gets learned. If I replicated that,   00:53:13.600 --> 00:53:18.480 could a transformer block kind of do  what the cortical columns are doing? 00:53:19.760 --> 00:53:23.680 Could I just instrument all my sensory  patches with little transformer blocks   00:53:23.680 --> 00:53:27.520 and then wire them up in the  right way and have it work? 00:53:27.520 --> 00:53:33.040 AHMAD: I think there'll be … there's still a  couple of things we need. One is that cortical   00:53:33.040 --> 00:53:38.880 columns are fundamentally sensory motor. And so  they're actually, each one, each cortical column   00:53:38.880 --> 00:53:46.640 is initiating actions, as well. So you cannot have  a static dataset fundamentally ahead of time. It's   00:53:46.640 --> 00:53:53.752 always a dynamic because we're constantly making  movements to get the next bit of data. And so … 00:53:53.752 --> 00:53:55.840 BURGER: Couldn’t I tokenize that, though? 00:53:56.880 --> 00:54:02.480 AHMAD: I mean, you could tokenize the input  and you can tokenize the output, but, you know,   00:54:02.480 --> 00:54:07.120 if you were to play the same set of inputs  back again to a network that … a cortical   00:54:07.120 --> 00:54:11.920 column that’s randomly wired differently, it  may make a different set of actions. And so   00:54:11.920 --> 00:54:16.880 as soon as it makes the first action that's  different, that dataset is no longer valid,   00:54:16.880 --> 00:54:23.360 right? It's, you know, there is ... you can't  fundamentally … you have to have a simulation   00:54:23.360 --> 00:54:28.960 of an environment rather than a static  one-way dataset, if that makes sense. 00:54:30.000 --> 00:54:36.160 So I think that's one piece that I think’s  missing in transformers today, is this,   00:54:36.160 --> 00:54:42.376 sort of, sensory-motor loop. And then the other  piece we talked about is continuous learning. 00:54:42.376 --> 00:54:42.901 BURGER: Yeah. 00:54:42.901 --> 00:54:44.952 AHMAD: I guess you said take  it off the table, but … 00:54:44.952 --> 00:54:46.137 BURGER: It's fundamental. 00:54:46.137 --> 00:54:49.920 AHMAD: Fundamental … different. Yeah, yeah.  And maybe one other difference. We talked,   00:54:49.920 --> 00:54:53.840 you know, much earlier about a single latent  space and the prediction that's being made   00:54:53.840 --> 00:54:58.800 at the top of the transformer that you compute  the loss function, and that's back-propagated   00:54:58.800 --> 00:55:04.880 through the transformer. That's not how  neurons learn. Neurons are making … every   00:55:04.880 --> 00:55:09.520 neuron is actually making predictions,  and every neuron is getting its input. 00:55:09.520 --> 00:55:13.680 And it's learning independent of  anything that happens at the top.   00:55:13.680 --> 00:55:18.640 And so it's a much more granular learning  signal. And information does flow from the   00:55:18.640 --> 00:55:22.160 top to bottom. But there's also many,  many other sources of information that   00:55:22.160 --> 00:55:28.400 it's learning from. So it's different in  that sense, as well, mechanistically. 00:55:28.400 --> 00:55:33.520 BURGER: The reason I ask, and now I'd like  to get into, you know, some of the ... the   00:55:33.520 --> 00:55:38.320 fun speculation because I've just ... it's been a  phenomenal discussion with the two. I think we've   00:55:38.320 --> 00:55:42.720 kind of elucidated the differences. Something I've  wondered after I've talked to both of you … and,   00:55:42.720 --> 00:55:47.680 you know, Nicolò, kind of learning about  this compression view of the world, lossless   00:55:47.680 --> 00:55:51.600 compression, and, Subutai, just, you know,  the Thousand Brains Theory and these cortical   00:55:51.600 --> 00:55:59.520 columns and the sampling of, you know, the world  to capture the signal that you can learn from. 00:55:59.520 --> 00:56:05.280 So let's say that I was able to design a  really small, efficient digital cortical   00:56:05.280 --> 00:56:10.880 column. Maybe it's transformer-based with  some, you know, a sparse representation   00:56:10.880 --> 00:56:15.520 and some sensory-motor mechanism built in.  Maybe it's more dendritic-based, you know,   00:56:15.520 --> 00:56:26.400 mapped into digital hardware. And I put a cortical  column on every sensor I have in the world,   00:56:27.280 --> 00:56:33.120 associated with every person, and wire them  up together with some of this and then have a,   00:56:33.120 --> 00:56:37.120 you know, billions of them that can  form higher-level abstractions. Like,   00:56:37.120 --> 00:56:40.360 what do you think would  happen? What could we do? 00:56:40.360 --> 00:56:47.120 AHMAD: That's a fantastic thought exercise,  I think [LAUGHS]. You know, again,   00:56:47.120 --> 00:56:52.640 assuming the cortical column is faithful and can  generate, you know, or suggest motor actions,  

00:56:52.640 --> 00:57:00.240 as well. I mean, in some sense, you could  potentially have a super intelligent system,   00:57:00.240 --> 00:57:04.960 right, that's far more intelligent  than anything else on the planet. 00:57:04.960 --> 00:57:11.511 Now we're scaling the number of cortical columns,  you know, not from a mouse, you know, to a hundred   00:57:11.511 --> 00:57:17.760 thousand columns that a human might have, but  potentially billions of cortical columns and way   00:57:17.760 --> 00:57:24.960 more. And there's no reason to think there's any  fundamental limit there. So this sort of a system   00:57:24.960 --> 00:57:29.680 is, I think, the way that superintelligent  systems will eventually be built. 00:57:29.680 --> 00:57:33.149 BURGER: But this is a very  different direction … 00:57:33.149 --> 00:57:33.703 AHMAD: It’s a very different ... 00:57:33.703 --> 00:57:35.520 BURGER: … than the one we're  currently headed down with,   00:57:35.520 --> 00:57:41.200 like, these monolithic models where we're  doing tons of RL, you know, to capture,   00:57:41.920 --> 00:57:46.320 you know, to get high-value human  collaboration in distribution. 00:57:46.320 --> 00:57:51.600 AHMAD: Yes. It's completely different  than the direction we're proceeding. 00:57:51.600 --> 00:57:57.120 So I think they, you know, to go down that path,  there needs to be a fundamental rethinking of   00:57:57.120 --> 00:58:02.400 some of our assumptions, potentially even  down to the hardware architectures that are   00:58:02.400 --> 00:58:05.920 necessary to implement it. The, you  know, fundamental learning algorithms,   00:58:05.920 --> 00:58:09.600 the fundamental training paradigm. We talked  about, you know, you can't have a static   00:58:09.600 --> 00:58:13.600 dataset. You're constantly moving around in  the world and doing things. So it's a very,   00:58:13.600 --> 00:58:18.160 very different way of going about  AI than what we're doing today. 00:58:18.160 --> 00:58:21.760 BURGER: Sounds like a great  time to be an AI researcher. 00:58:21.760 --> 00:58:24.872 AHMAD: Absolutely. [LAUGHTER] 00:58:24.872 --> 00:58:29.200 BURGER: Nicolò, what was your  reaction to that hypothesis? 00:58:29.920 --> 00:58:33.840 FUSI: It sounds super interesting. I  mean, my brain was churning. You know,   00:58:34.960 --> 00:58:41.600 my background is very different. And so, like, I'm  in a much worse position to answer this question.   00:58:41.600 --> 00:58:46.640 But I was starting to think, OK, so let's say  I do this. What would be my loss function?   00:58:47.840 --> 00:58:51.680 What, you know, how would information  flow through the system? Like,   00:58:51.680 --> 00:58:54.400 sounds like cortical columns would  each have their own loss that then   00:58:54.400 --> 00:58:59.280 I would aggregate—and then I would add a  contribution that is, like, higher level. 00:58:59.280 --> 00:59:01.120 And then back to

my question. You know,   00:59:01.120 --> 00:59:08.560 how is the temporal information coordinated?  Because one way to see this is that, you know,   00:59:08.560 --> 00:59:12.320 the way I'm coming to understand this is that  it's kind of like a multi-view framework. 00:59:12.320 --> 00:59:18.640 You have the same phenomena represented to  multiple independent, but at the same time, views.   00:59:18.640 --> 00:59:24.240 And so part of me is like it feels like that  you need to tie together these cortical columns   00:59:24.240 --> 00:59:28.160 in such a way that they all get that gradient  feedback if you're training with gradient-based   00:59:28.160 --> 00:59:34.000 methods, for instance. And so that's, kind  of, it feels super, super interesting. 00:59:34.000 --> 00:59:38.160 It is related to a lot of, you know, very  superficially, to a lot of ideas in machine   00:59:38.160 --> 00:59:42.400 learning around, hey, is it better to have  one giant super deep network? Is it better   00:59:42.400 --> 00:59:46.880 to have a bunch of shallow networks? But the  difference is also in the way you train them,   00:59:46.880 --> 00:59:50.960 right? We typically train this bunch  of shallow networks on kind of the   00:59:50.960 --> 00:59:54.320 same objective and the same  data and not typically into   00:59:54.320 --> 01:00:01.200 an experiential cycle. Whereas this sounds  like this is a different way to do it. 01:00:01.200 --> 01:00:08.480 BURGER: Right, right. I think … I want to  pull this back around to the title of the   01:00:08.480 --> 01:00:16.400 podcast. And so I'll share an observation.  You know, so I've been using some of the   01:00:16.400 --> 01:00:20.960 latest models to code. You know, they're  getting better really fast. I've been using   01:00:20.960 --> 01:00:24.880 them to kind of relearn some of the physics  that I never really understood deeply. 01:00:24.880 --> 01:00:30.480 You know, especially in general relativity,  like E=MC2. Like, why is C in there at all,   01:00:30.480 --> 01:00:34.160 right? Just stuff like that. Because  now it can actually explain it to me,   01:00:34.160 --> 01:00:38.560 and I can keep beating at it until I  understand it, and then, of course, work. 01:00:38.560 --> 01:00:45.040 And at some point, I asked the model, “Can you  describe how I think?” And I was just curious.   01:00:45.040 --> 01:00:51.840 And it, you know, it gave me a page description  that my jaw dropped because I said this, this   01:00:52.720 --> 01:00:58.240 thing knows me better than I know myself.  I don't think any human being, including me,   01:00:58.240 --> 01:01:03.040 could have captured kind of the way my approach  to learning and my brain works, and I just   01:01:03.040 --> 01:01:06.880 read it as, like, like, yep, that's right. And  I learned something about myself.  01:01:08.240 --> 01:01:11.920 So I wouldn't say that it passed the Turing  test because this is way beyond Turing test.   01:01:11.920 --> 01:01:15.520 This was like, this thing knows me way  better, you know, than I thought any   01:01:15.520 --> 01:01:19.600 machine ever could. I mean, I'm having a  conversation with it. It could be human,   01:01:19.600 --> 01:01:30.880 but it's superhuman. So in some sense, it's  like intelligent beyond human capabilities   01:01:30.880 --> 01:01:36.720 with its ability to discern patterns in  how someone's interacting.  And yet it's a   01:01:36.720 --> 01:01:41.840 tool. You know, it's not conscious.  It doesn't have agency, embodiment,   01:01:41.840 --> 01:01:47.120 emotion. It understands a lot of that stuff  from the training data. But at the end of the   01:01:47.120 --> 01:01:51.600 day, it's a stochastic parrot, right? It's got,  you know, it's got the weights, and I give it   01:01:51.600 --> 01:01:55.440 a token, and it outputs a token. So, like,  are these machines intelligent or not?  01:01:55.440 --> 01:02:01.513 FUSI: I’ll let Subutai answer first. [LAUGHS] 01:02:01.513 --> 01:02:10.640 AHMAD: OK. You know, you know, it's definitely  a savant, right? It knows a huge amount about   01:02:10.640 --> 01:02:15.200 the world. It's absorbed a lot of stuff,  and it can articulate that in ways that   01:02:15.200 --> 01:02:21.360 are just amazing. And, you know, it's  taken your chat history with, you know,   01:02:21.360 --> 01:02:26.480 presumably thousands of chats and able to  summarize that in a way that's remarkable. 01:02:27.200 --> 01:02:33.440 At the same time, I think, you know,  transformers are not intelligent in   01:02:33.440 --> 01:02:39.120 the way that a three-year-old is, right?  A three-year-old human is very curious, is   01:02:39.120 --> 01:02:46.000 constantly learning. It can learn almost anything.  And, you know, a three-year-old Einstein was able   01:02:46.000 --> 01:02:51.280 to learn and eventually come up with theories  that shook the world. That, you know, E=MC2. 01:02:53.520 --> 01:02:58.240 And so, you know, could a transformer do  that? I don't think so. And so I think   01:02:58.240 --> 01:03:05.280 there's still a difference. There's things  it can do that are amazing. But there are   01:03:05.280 --> 01:03:10.560 still basic things that a child can do that  transformers cannot do. So I think there's   01:03:10.560 --> 01:03:17.040 still a gap there. Exactly how to articulate  it, and how to bridge that gap, is, of course,   01:03:17.040 --> 01:03:21.160 the trillion-dollar question. But it is  bridgeable. And there is a gap today. 01:03:21.160 --> 01:03:24.080 BURGER: Right. Nicolò? 01:03:24.080 --> 01:03:32.400 FUSI: You know, I think, from my perspective,  they are intelligent. And from my perspective,   01:03:32.400 --> 01:03:34.720 I go back to the definition  of intelligent, which is like,   01:03:34.720 --> 01:03:40.960 can you achieve your objectives in a variety  of environments? It's a very basic fundamental,  

01:03:40.960 --> 01:03:45.200 but it's kind of, you know, it can be  embodied, a form of embodied intelligence,   01:03:45.200 --> 01:03:50.080 an agentic intelligence. If I plop you in  an environment, and I give you an objective,   01:03:50.080 --> 01:03:57.520 can you achieve it? And the wilder the  environment, the harder the task is. 01:03:57.520 --> 01:03:59.680 And I do think … I agree with Subutai. Like,   01:03:59.680 --> 01:04:02.057 there is a jaggedness of  intelligence we keep describing. 01:04:02.057 --> 01:04:02.578 BURGER: Yup. 01:04:02.578 --> 01:04:07.680 FUSI: Like these things cannot be simultaneously  super good, you know, Olympiad-level   01:04:07.680 --> 01:04:14.240 mathematicians and still give you stupid answers  when you're trying to, I don't know, you know,   01:04:14.240 --> 01:04:19.983 figure out which cable goes where in your … in  your car's battery, you know, like, whatever. 01:04:19.983 --> 01:04:22.720 BURGER: [LAUGHS] Well, then it's better than  me. I'm not an Olympiad-level mathematician,   01:04:22.720 --> 01:04:24.280 and I do stupid stuff all the time. 01:04:24.280 --> 01:04:28.320 FUSI: I know exactly. Well, you know,  whatever that was, that was a bad example.   01:04:28.320 --> 01:04:34.320 But you get it. But part of it goes back to  the compression view. Like, I do believe that   01:04:35.360 --> 01:04:39.600 intelligence is compression. So the ability to  come up with succinct explanations for complex   01:04:39.600 --> 01:04:46.480 phenomena and even succinct explanations for  complex worlds, and then it implies or leads   01:04:46.480 --> 01:04:51.360 to your ability to operate within them, and the  fact that we have these things that they can   01:04:51.360 --> 01:04:59.040 prove crazy theorems but at the same time fail  at fairly rudimentary tasks is a sign that the,   01:04:59.040 --> 01:05:03.360 yes, transformers are great in terms of  inductive biases they put on the world and   01:05:03.360 --> 01:05:09.600 computation that are great, but we're ultimately  all subject to the No Free Lunch Theorem. 01:05:09.600 --> 01:05:14.160 You know, across the world, the set  of tasks that you could be pursuing.   01:05:14.880 --> 01:05:20.320 You know, you have certain inductive biases that  kind of privilege certain tasks at the expense   01:05:20.320 --> 01:05:25.120 of others. And there isn't, like, a thing  yet that has expanded our set of tasks that   01:05:25.120 --> 01:05:31.920 are addressable. And so I do think that it's a  matter of rethinking our approach to a few things,   01:05:31.920 --> 01:05:39.040 whether I think likely both on the architecture  front and on the losses and the way we train these   01:05:39.040 --> 01:05:45.200 systems front. I think there is an opportunity  to expand the intelligent frontier of these   01:05:45.200 --> 01:05:51.680 models. But yeah, from my perspective, they are  intelligent already just in a jagged way. 01:05:51.680 --> 01:05:55.120 BURGER: It's such an interesting question, and  I know a lot of people write a lot about this,   01:05:55.120 --> 01:05:59.120 so I don't think treading any  new ground here. But, you know,   01:05:59.120 --> 01:06:06.880 there's the diversity of the tasks you can excel  at. You know, are you able to handle nuance and   01:06:06.880 --> 01:06:12.560 understand things deeply? Are you able to learn  continuously? Right now, the systems can't,   01:06:12.560 --> 01:06:17.200 right. Are you embodied? I don't know if  that matters. Do you have an objective? Well,   01:06:17.200 --> 01:06:21.600 we could give them one. Are you conscious? Is  that … I mean, that's a whole other thing. 01:06:21.600 --> 01:06:30.160 So it just feels like there's a bunch of check  boxes, and we've checked a bunch of them, and a   01:06:30.160 --> 01:06:36.480 bunch of them are unchecked. And maybe there's  no consensus on, like, where that threshold is   01:06:36.480 --> 01:06:41.200 because there are many dimensions of intelligence,  and some of which humans don't even have. 01:06:41.200 --> 01:06:46.080 FUSI: And that's why we have the term AGI and ASI,   01:06:46.080 --> 01:06:51.360 and people are debating the G and the S—what  is general, what is specialized. So there is,   01:06:51.360 --> 01:06:56.800 like, it's a huge discourse, like, for sure.  But that's why we had to start characterizing.   01:06:56.800 --> 01:07:00.720 But if you go back in the definition,  going back to my schooling, go back to   01:07:00.720 --> 01:07:05.840 the definition of intelligence from Plato and  Aristotle and Descartes, like, in some sense,   01:07:05.840 --> 01:07:10.615 you see the goalpost moving through the centuries  around what we define as intelligent. 01:07:10.615 --> 01:07:12.320 BURGER: Right. FUSI: And I feel like we are still doing it. 01:07:12.320 --> 01:07:14.880 BURGER: Yeah. We’ll be doing  it for a long time, you know,   01:07:14.880 --> 01:07:18.880 which in AI velocity is probably  another like four or five years. 01:07:18.880 --> 01:07:23.600 Hey, I just want to thank you  both for the dialogue. You know,   01:07:23.600 --> 01:07:30.080 I treasure both of you as, you know,  intellects and scholars and friends.   01:07:31.040 --> 01:07:37.480 It was just a joy to nerd out with you all.  So thank you both for taking the time. 01:07:37.480 --> 01:07:40.000 AHMAD: Thank you so much, Doug, for having me. 01:07:40.000 --> 01:07:45.811 FUSI: Thank you for having us. This was great. 01:07:45.811 --> 01:07:45.824 [MUSIC] 01:07:45.824 --> 01:07:46.355 STANDARD OUTRO: You’ve been listening  to The Shape of Things to Come,   01:07:48.357 --> 01:07:51.440 a Microsoft Research Podcast.  Check out more episodes of the   01:07:51.440 --> 01:08:05.840 podcast at aka.ms/researchpodcast or on  YouTube and major podcast platforms. 01:08:05.840 --> 01:08:06.623 [MUSIC FADES]