WEBVTT
1
00:00:00.120 --> 00:00:09.180
So we, you know, set out looking for interesting examples, and one of them was like weather modeling because the weather data is open source.
2
00:00:09.320 --> 00:00:16.900
And so given that the data was there, we were like, "Okay, let's just go try it," right? Yeah. And that's the beauty of it whenever data is available, it's really good news.
3
00:00:17.020 --> 00:00:23.500
But a lot of weather scientists did caution us back then. This was back in '21, and they said, "No, no, no, this is so difficult."
4
00:00:23.880 --> 00:00:32.240
You know, there have been decades of development in traditional weather forecasting, and that's very careful, bottom-up, physics-based modeling, right?
5
00:00:32.280 --> 00:00:39.040
So assuming, oh, this is the fluid dynamics, can you go predict the weather, uh, the next day, and so on.
6
00:00:39.380 --> 00:00:48.640
And so that's how a lot of the thinking was that AI is just not gonna be able to beat, uh, the decades of work in weather modeling.
7
00:00:49.140 --> 00:01:08.760
Uh, but to our surprise, we just went ahead, we trained them, we used neural operators to be able to effectively capture the phenomena, and then we found that it's not only accurate, it's almost as close to what the traditional weather models can do accurately, but also tens of thousands of times faster.
8
00:01:09.320 --> 00:01:22.060
So what would take a big supercomputer to run can now be run, and we only needed a consumer-grade like GPU. Like, you know, it was a small model. It fit very well. It's very fast, and it's accurate.
9
00:01:22.380 --> 00:01:35.940
And I think that just changed everybody's thinking. Welcome to Latent Space. Um, this is the AI for Science section of Latent Space. Um, I'm Brandon. I work on RNA therapeutics using AI at Atomic AI.
10
00:01:36.320 --> 00:01:44.040
I'm joined by my co-host, RJ Haneke, who develops, uh, uh, spatial transcriptomics and is the CTO and founder of Mira Omics.
11
00:01:44.460 --> 00:01:52.580
Uh, today, we're excited to be joined by Anima Anandkumar, uh, the Brin Professor of, uh, Mathematics and Computer Science at Caltech.
12
00:01:52.940 --> 00:02:03.960
Um, Anima has done all sorts of really cool work, uh, combining AI with basically models of the physical world, um, and has a really diverse background. I don't think I could even remotely cover it.
13
00:02:04.520 --> 00:02:10.199
Um, but anyway, I'll let Anima introduce herself. Thank you for coming on the show. Yeah. Yeah. Thank you, Brandon and RJ.
14
00:02:10.240 --> 00:02:22.140
It's a pleasure to be there, and I really like the term latent space because that very much [chuckles] figures in a lot of my work, uh, because it's really, you know, the world is latent.
15
00:02:22.840 --> 00:02:37.600
Uh, but yeah, just as a brief introduction, uh, you know, I've been working in AI for more than two decades, uh, in a way, you know, before even deep learning when a lot of the theoretical foundations had to be built for probabilistic models, I worked on them.
16
00:02:37.980 --> 00:02:44.320
And then as deep learning started taking off, I also had a foot in industry until recently. So I was at NVIDIA.
17
00:02:44.420 --> 00:02:56.080
I led AI research there, and before that, at Amazon Web Services, helped found the cloud AI team and s- built the first cloud for AI products, uh, uh, back almost a decade ago.
18
00:02:56.500 --> 00:03:11.940
So, you know, like kind of having this one foot in industry and academia, I think has given me a lot of interesting perspective of how to bring theory and practice together and think of AI at large scale, but also AI that is principled.
19
00:03:12.440 --> 00:03:25.440
A lot of your work has been related to the modeling of physical systems, um, using s-certain types of physical systems, uh, which you model with differential equations, and you, uh, help model them with using machine learning.
20
00:03:25.540 --> 00:03:35.940
So, um, maybe first let's go in and talk a little bit about that as a high level, but we'll get to kind of the details about using neural operators and some of the applications like weather later.
21
00:03:36.460 --> 00:03:46.000
But first, I'm actually really curious to hear about TorchLean and how this, uh, recent work you've been doing connects with that, um, larger research program. Mm-hmm.
22
00:03:46.380 --> 00:03:53.300
To me, broadly, like, you know, my thesis is AI and science, how we bring that together, right?
23
00:03:53.460 --> 00:04:10.280
So, uh, you know, when I started at Caltech almost a decade ago, that's when, you know, my passion was always science, was physics, and but, you know, I was doing AI, so how to bring that together, uh, was where, you know, the first kind of foundations got laid there.
24
00:04:10.860 --> 00:04:22.820
And to me, like, you know, there are several aspects to that. One is people have been thinking how to use language models for science. Yes, you can do a lot of hypothesis generation.
25
00:04:22.900 --> 00:04:34.500
You can have ideas, but ideas are not enough, right? So you can have a lot of ideas. The bottleneck is going, testing, and verifying that they work in the real world.
26
00:04:34.940 --> 00:04:51.360
And so this aspect is where a lot of my recent focus has been on how do we ensure that we can build AI that has guarantees that it will work in the physical world or any aspects in scientific domains.
27
00:04:51.860 --> 00:05:07.420
And, and one way to think about it is, you know, can we model the physical world and keep the physics correct? And that's where neural operators come in. The other aspect is can we verify symbolically certain aspects?
28
00:05:07.580 --> 00:05:19.580
Uh, for instance, you know, if we, uh, claim that the theorem is correct, we have to go verify that. You know, that's where Lean as a formal language can be useful for verification.
29
00:05:19.960 --> 00:05:28.340
So how do we bring that together with language models is where a lot of mathematical reasoning has been at the forefront.
30
00:05:28.760 --> 00:05:43.300
And so TorchLean kind of is in that realm where we say, you know, not only that you wanna verify mathematical statements, you may wanna verify s- uh, what neural networks themselves claim to deliver.
31
00:05:43.700 --> 00:05:50.040
You know, for instance, if you're now using a neural network and you wanna ask whether it's gonna be robust.
32
00:05:50.240 --> 00:06:00.220
Say you wanna use a neural network in a control loop, or you wanna control, um, you know, whether it's a drone, whether it's, uh, a nuclear reactor.
33
00:06:00.540 --> 00:06:14.060
So all of this ultimately when we br- build AI systems with deep learning into control loops, we want robustness. And so now TorchLean can help us do those verifications seamlessly.
34
00:06:14.520 --> 00:06:22.320
So we can now have neural networks be part of the verification loop and have confidence that we can use them appropriately.
35
00:06:22.940 --> 00:06:36.088
We have already discussed on the podcast Lean and, uh, and ev-everyone should probably be familiar with neural networks. How- But neural networks seem very unconstrained. What kinds of proofs are you talking about?
36
00:06:36.148 --> 00:06:44.868
Are you talking about bounds on the outputs, inputs? Um, what, what can you prove with- Yeah... Torchlean? Yeah. So Torchlean is a, an overall framework, right?
37
00:06:44.948 --> 00:06:51.228
So what it really enables is that you can now write neural networks essentially in Lean.
38
00:06:51.728 --> 00:07:04.008
So instead of writing in like PyTorch, it's like a PyTorch-like abstraction, but you can like kinda, you know, write it in Lean, and so it can be fully formalized in Lean. And then there are several implementations.
39
00:07:04.068 --> 00:07:14.188
You know, we have algorithms for certified robustness like Crown. You know, those are implemented under this framework. So- Sorry, what was it? Like what? Like Crown? Crown.
40
00:07:14.368 --> 00:07:25.548
Crown is one of the, um- Can you explain a little bit? So there are different ways to bound, you know, for certified robustness. You know, how tight those bounds can be, it depends on the relaxation techniques.
41
00:07:26.028 --> 00:07:37.608
And sort of without going into those, there's many such algorithms. But, you know, we are kinda like implementing them and enabling them in Lean, so we can seamlessly run both.
42
00:07:37.868 --> 00:07:50.588
You know, we can both first kinda write down in a Torch-like framework neural networks very simply, right? And then we can also make statements about them formally and verify them.
43
00:07:50.688 --> 00:08:01.488
So all of that can be brought together in one framework. So what's an example of a, a bound that you could claim? Like, so we're operating a nuclear reactor, we don't want it to melt down.
44
00:08:01.768 --> 00:08:13.608
What are the sort of guarantees that, um, you could provide to the out, inputs and outputs that would help that not melt down? Yeah. I mean, the natural one is the certified robustness that I mentioned.
45
00:08:13.688 --> 00:08:25.388
So saying that if you-- our inputs are, you know, perturbed by a certain amount, how much is the output gonna be perturbed, right? The sensitivity analysis, uh, is another term.
46
00:08:25.748 --> 00:08:41.648
And so having those kinds of bounds for different neural architecture, so you kinda automatically get those bounds, can then help us, you know, not only train neural networks to do well in a control loop, but also worry about safety and robustness or stability.
47
00:08:41.768 --> 00:08:47.368
These are all part of control systems that people worry about, so that's one example of an application.
48
00:08:47.888 --> 00:08:58.148
So it's really more broadly the idea is you need verification, uh, in lots of scenarios that involve neural networks. So control loops are one.
49
00:08:58.548 --> 00:09:12.328
Another example is, you know, we used physics-informed neural networks to, say, solve partial differential equations or come up with systems that gua-- that are guaranteed to satisfy certain physical laws.
50
00:09:12.788 --> 00:09:24.868
But we also wanna verify, for instance, that our neural network is only trained in finite precision, right? So can we overcome those requirements, and what happens when we, uh, are...
51
00:09:25.348 --> 00:09:34.408
What are the shortcomings because we are using this finite precision? Can we also bound those? Oh, right. So those are other kinds of bounds that work in Torchlean.
52
00:09:34.468 --> 00:09:50.728
So all aspects of like, uh, you know, the effect of precision, the effect of perturbation, all of these we can, uh, you know, we can have algorithms that are implemented in Lean that can be seamlessly now part of the verification loop.
53
00:09:50.888 --> 00:10:00.468
So, and do-- is the descriptive power of the Torchlean, is that sufficient to describe basically any neural network, or is there, are there constraints on that? Yeah.
54
00:10:00.528 --> 00:10:10.588
So it's essentially a, you know, PyTorch-like, you know, framework, right? So you can just kinda nicely define neural net layers in the same way.
55
00:10:11.148 --> 00:10:23.388
Uh, but the back end having like Lean, uh, helps us formalize and prove it. And for like transformer architecture, for example, is it reasonable to prove these kinds of bounds on a very large neural network?
56
00:10:23.448 --> 00:10:34.508
So the- Or transformer. You know, there is the aspect of one is like kinda having the framework, right? The other is scalability. Lo-- So Lean still has a lot of, uh, shortcomings there.
57
00:10:34.588 --> 00:10:47.948
It's CPU-based, and, uh, you know, it's not, uh, like make getting that onto the GPU has a lot of- Yeah... nuances there. Uh, so, you know, a lot of work needs to be done. So what we've started with is a framework.
58
00:10:48.047 --> 00:10:57.188
You know, making that more efficient, especially at a very large scale, requires still a lot of work to be done. Yeah. But that's true broadly for Lean as well.
59
00:10:57.848 --> 00:11:00.568
And just, uh, trying to understand like how we imagine this.
60
00:11:00.608 --> 00:11:13.908
So if I were to take a, say, numerical analysis classes, you know, on graduate level numerical analysis class, you have a differential equation, you have some discretization error or something, and you bound like given these properties, I can bound the solution.
61
00:11:14.228 --> 00:11:24.028
Mm-hmm. Right? So solving, um, some of these physics-based, uh, or AI-based solutions to differential equations, um, I think historically has been kind of the wild west.
62
00:11:24.088 --> 00:11:30.168
So I think you mentioned physics-inspired neural networks. Um, really cool idea. It'd be fun to talk about that a little bit.
63
00:11:30.728 --> 00:11:38.968
But I, I know that sometimes they, they are particular, and that people don't-- they don't always work, and I think people don't always know when they will or won't work.
64
00:11:39.328 --> 00:11:51.828
Um, I, I mean, I'm not an expert, but I'm, I'm just wondering if that's, um, in your experience. Um, and what I'm wondering is like, has this helped you understand like w-- the domain of applicability for PINNs or...
65
00:11:52.368 --> 00:12:04.168
And is that sort of like the goal is like you can rigorously say like, "This solution will converge," or is there not necessarily the same concepts of convergence in a controlled way for neural networks? Yeah.
66
00:12:04.228 --> 00:12:18.288
So, you know, like physics-informed neural nets are about like saying that, you know, I write down like a PDE, uh, partial differential equations, and hopefully the optimization succeeds and I get the answer, right?
67
00:12:18.668 --> 00:12:28.368
And of course, if like optimization was not at all an issue, this would be universal. You solve everything, you know, we're all happy, but that's not the case.
68
00:12:28.748 --> 00:12:45.488
And so optimization ends up being usually very difficult, especially for problems that are time-dependent, meaning it's not just stationary, you also have time, and the time component in many cases could be turbulent, like in the case of fluid dynamics.
69
00:12:45.908 --> 00:12:57.224
You know, you kinda like if you- Run it long enough, uh, it can become chaotic, so you really, you know, have, like, very small fine-scale effects matter.
70
00:12:57.864 --> 00:13:06.344
And so in those cases, just trying to solve a partial differential equation at all times, uh, is just hopeless.
71
00:13:06.444 --> 00:13:13.504
Like, you know, this is not an optimization landscape that, uh, you know, I think we'll, you know, we can have any handle on.
72
00:13:13.884 --> 00:13:24.724
And this is where the idea that from scratch we would be able to solve these, uh, equations using a neural net is not possible, so PINNs don't work everywhere.
73
00:13:25.304 --> 00:13:38.624
And our idea of neural operators came as a way to overcome this, right? So saying, you know, we can't rely just on physics constraints alone to come up with answers. We have lots of data available.
74
00:13:38.864 --> 00:13:51.184
Um, you know, I'll talk about the weather example where we even collect data, right? So we don't just solve equations and g- have synthetic data, but we also have real data by observing the weather as one example.
75
00:13:51.584 --> 00:14:07.704
So why not make use of all of the data available so we don't just rely on trying to solve partial differential equations and other physical problems from scratch, because it's really the data-driven approach that makes it possible to get quick answers.
76
00:14:08.084 --> 00:14:21.244
And so with neural operators, we can bring both of them together. We can have all the data that's available. We can utilize it. We can add physical constraints, and then that overcomes the limitations that PINNs face.
77
00:14:21.844 --> 00:14:27.244
Can you give a little bit more intuition on the difference there and why that is possible?
78
00:14:27.324 --> 00:14:39.924
So I heard you mention, you know, in, with PINNs, you're basically just baking the physics constraints into the neural network, but that this becomes unstable over time or other, other variables.
79
00:14:40.444 --> 00:14:49.104
Whereas if you add a little bit of data, like I, I can kind of un- intuitively understand why that might help, but can you give a little intuition for what's going on? What's the difference here?
80
00:14:49.584 --> 00:14:57.604
So with the PINN, like, you know, every instance of an equation you solve from scratch, right? At least in the classical sense.
81
00:14:58.044 --> 00:15:08.064
So you start, you take s- the specification of what equation you want to solve, and you hope that the optimization landscape succeeds, which many cases it doesn't.
82
00:15:08.524 --> 00:15:20.144
Whereas with the neural operators, what we do is we, you know, have lots of tr- data, so we have a training phase. Uh, we teach it how to come up with solution for different instances of equation.
83
00:15:20.684 --> 00:15:29.824
And so just as in other supervised learning, at test time, you can now ask, you know, "Can you come up with an answer?" And you can still have physics constraints as a way to guide that.
84
00:15:30.404 --> 00:15:41.904
Uh, so, you know, it can be both data-driven and physics informed together. Uh, but the benefit is because we have data, you know, it's m- like you're not stuck in an optimization landscape, right?
85
00:15:41.984 --> 00:15:50.744
So you know what the answers are during training, so you're now at a better chance to come up with the right answers even at test time.
86
00:15:51.184 --> 00:16:05.864
My understanding is that a neural operator is, is, uh, a function fit to, to, uh, like, to data or a neural network, you know, learns to fit functions to data. Is that a good intuition here? Yeah.
87
00:16:05.924 --> 00:16:13.244
So, you know, neural operators are, in that sense, similar to... You know, it's the same as neural networks, right? You're learning on data.
88
00:16:13.604 --> 00:16:24.764
But the difference is neural operators are, you can think of it as a generalization of neural networks. So with standard neural networks, the inputs and outputs are of fixed sized.
89
00:16:25.204 --> 00:16:29.344
So in language, we have fixed vocabulary, we fix what the input and output are.
90
00:16:29.744 --> 00:16:41.544
And same with images, uh, in computer vision and videos, we assume a fixed resolution, and we always, you know, our inputs and outputs are always at that fixed resolution. We can't change it post-hoc.
91
00:16:42.104 --> 00:16:52.184
Whereas with a lot of this physical data, the idea is our world is inherently multi-scale, so you should not be, like, deciding beforehand what the resolution is.
92
00:16:52.624 --> 00:17:01.344
You know, maybe you have, like, weather data available only at coarse resolution, but really the actual phenomena is happening at a finer scale, right?
93
00:17:01.424 --> 00:17:10.864
And maybe you wanna after that incorporate, uh, either additional data at finer resolution or add in physical constraints at finer resolution.
94
00:17:11.304 --> 00:17:22.144
So we should be having that flexibility, and we should really think of the world not at these fixed resolution, but one that's happening, uh, infinitely [chuckles] you know, that at one...
95
00:17:22.624 --> 00:17:25.444
The real world happens at that infinite resolution.
96
00:17:25.864 --> 00:17:36.304
And that's what neural operators enable because they model inputs and outputs as continuous functions that can be infinitely resolved, that can have infinite discretization.
97
00:17:36.904 --> 00:17:45.804
And now we can have, uh, you know, at inference time, you can give it now inputs and, uh, ask for outputs at any resolution.
98
00:17:46.264 --> 00:17:59.144
So you're not just limited to the resolution of training that we see in standard neural networks, and that's what neural operators enable. So neural operators enable us to zoom in and out as we like.
99
00:17:59.744 --> 00:18:12.844
So obviously that, as stated, that any function that, that's under a constraint, right? You could have many, many functions that fit this, fit the data. It would be easy to overfit. So how do you regularize that? Yeah.
100
00:18:13.284 --> 00:18:25.844
Certainly, like, you know, uh, you know, if you're, uh, asking about making predictions at a higher resolution than what's seen, like what we call zero-shot super resolution, uh, you're kind of making some guesses, right?
101
00:18:25.924 --> 00:18:33.344
And, and that's what these models are doing. They're trying to regularize and kind of smoothly extend to higher resolution.
102
00:18:33.664 --> 00:18:51.164
But of course, if you now give it the model additional information in terms of, let's say, a physical loss, so you could give it pa- partial differential equation constraints, conservation laws, and you can now enforce them at a finer resolution than the data you have, then there's more guidance in a way.
103
00:18:51.644 --> 00:19:02.104
So that way it can now come up with the, uh, the right answers even at higher resolution because you're, you know, giving it constraints at higher resolution.
104
00:19:02.164 --> 00:19:13.224
And so that's how we can ensure that these physics informed neural operators can work at higher fidelity and higher resolution than even the training data that was available.
105
00:19:13.840 --> 00:19:30.180
My understanding is a, a lot of your work uses a particular kind of, uh, neural operator, a Fourier neural operator. So Fourier is a dual domain, it is extended across the entire, uh, domain of the inputs.
106
00:19:30.740 --> 00:19:37.960
That's a lot of jargon, maybe. Can you give some intuition for why, why is that important? How does that help?
107
00:19:38.500 --> 00:19:50.540
What I mentioned neural operators as a class of, uh, mo- models that allow us to have any resolution input and any resolution output, right? And learns the mapping between them.
108
00:19:50.820 --> 00:19:58.240
You know, that's really called an operator. So [chuckles] the mapping between function spaces. So that's the reasoning behind the name neural operator.
109
00:19:58.760 --> 00:20:07.100
And, uh, you know, Fourier neural operator was one of the early, uh, setups we y- or architectures we came up with.
110
00:20:07.540 --> 00:20:20.980
And the reason why that's been so successful is because it kind of strikes a nice trade-off between efficiency and expressivity, right? So why is the Fourier space a good one?
111
00:20:21.420 --> 00:20:29.400
The Fourier space allows us to, you know, it's a dual space, like you mentioned, but it really allows us to capture non-local phenomena, right?
112
00:20:29.520 --> 00:20:36.860
So meaning something that's like, uh, a non-local in the Fourier domain could be even efficiently captured.
113
00:20:37.400 --> 00:20:48.700
And there are a lot of phenomena like we see in nature, whether it's fluid dynamics, material deformation, quantum chemistry, it's all, you know, there's a lot of them are non-local.
114
00:20:48.940 --> 00:20:57.360
You know, the differential equation, like the derivative is local, but the inverse of it is you're kind of doing essentially integration, it's non-local, right?
115
00:20:57.420 --> 00:21:03.160
So the solutions are non-local, and these models are able to capture that.
116
00:21:03.480 --> 00:21:15.800
But at the same time, doing Fourier transform is efficient, and it kind of like nicely captures a lot of inductive bias we see in many of these, um, natural phenomena.
117
00:21:16.120 --> 00:21:28.880
But this doesn't mean that we are capturing the world entirely in the Fourier sp- uh, basis, right? It's not a linear representation in the Fourier basis, which is what classical numerical methods do.
118
00:21:29.340 --> 00:21:38.760
We add non-linearity just as in transformer and other neural nets in between Fourier layers. And we also add residual collections.
119
00:21:38.820 --> 00:21:51.080
So all of these architectural aspects that are inspired by other neural nets that work well in other neural nets, bringing that together really kind of helps us get best of both the worlds.
120
00:21:51.120 --> 00:22:03.440
So you can think of like if we were to use transformers and we require a very high resolution, it would become untenable because of the quadratic complexity and all to all connections.
121
00:22:03.880 --> 00:22:17.520
On the other hand, if you did that with Fourier transforms, uh, we have like quasi-linear, uh, complexity, and still we have global connections in a way we can model these non-local phenomena.
122
00:22:17.860 --> 00:22:24.720
And so that's why it's a nice middle ground. So that allows you to learn from what is happening on the other...
123
00:22:24.760 --> 00:22:33.820
Like, it- it's talking weather, like what's happening in Chicago may have some impact on what's happening in San Francisco, or maybe not, but that's the idea. Yeah.
124
00:22:33.860 --> 00:22:46.680
So that's the idea in time, like kind of, yes, at this part may be local, but eventually they have an impact in other locations and, uh, yeah, so both in space and time, we want to capture that dependence.
125
00:22:46.840 --> 00:22:57.900
Yeah, so what happens today in Chicago will happen-- will have an impact in a month in San Francisco or something like that. Uh, yeah. So w- you know, so there is like both the short-term and the long-term effects.
126
00:22:57.940 --> 00:23:04.800
So in a short-term, like we think about predictable weather, but longer-term, we're talking about climate, right? Right.
127
00:23:04.900 --> 00:23:18.860
So what happens, you may not be able to say precisely, you know, what happens in Chicago, what will happen in Ch-San Francisco. Yeah. That's like the butterfly effect. On the other hand, we can kind of give averages.
128
00:23:18.920 --> 00:23:30.360
You know, if there's heat wave in this kind of overall region, you know, we kind of have an idea that it's gonna be higher than average temperatures. So those are the aspects we can capture together.
129
00:23:30.900 --> 00:23:48.960
From an architectural standpoint, for all the AI engineers here, are we just talking about doing all the work in the Fourier domain, but it's basically the same neural network, but I'm just operating in the Fourier domain, or is there other, other aspects that are cons- that are required in order to do this properly?
130
00:23:49.040 --> 00:23:53.620
So think of it, I guess the, maybe the easiest way to think about it is, you know, you can...
131
00:23:53.960 --> 00:24:03.220
If you think of a transformer architecture instead of like the, uh, you know, attention map, you now have the Fourier, but you still have other non-linearities.
132
00:24:03.320 --> 00:24:11.360
You have like, you know, the residual, you have, you know, many other parts of the architecture still there that give it like expressivity.
133
00:24:11.720 --> 00:24:23.180
And we have lifting to higher dimension, like, you know, in a channel space to give it more expressivity. So all of those kind of best principles are, are still available. Mm-hmm.
134
00:24:23.260 --> 00:24:34.040
Uh, but the Fourier helps us capture that all to all, uh, you know, dependence without requiring very huge complexity that we see in transformers. That makes sense.
135
00:24:34.560 --> 00:24:39.540
The other advantage is that it gives you the natural multi-scale with sort of a implicit cutoff.
136
00:24:40.180 --> 00:24:53.080
Is, um, you know, the sort of, if you have a signals background or physics background, you might ask, uh, you know, in linear, if you're doing everything linearly, there's a maximum frequency and you know, you know, above that you can't represent anything.
137
00:24:53.720 --> 00:25:03.620
Um, but how does that-- how does adding these other architectural changes in a non-linear domain actually affect, you know, your choices of frequency bounds and- Yeah. Yeah.
138
00:25:03.660 --> 00:25:07.520
No, that, that's a great question, and that's where the expressivity comes in, right?
139
00:25:07.620 --> 00:25:23.380
Otherwise, if you're just taking a Fourier transform of a signal and trying to represent it, you know, that's what numerical methods have also attempted to do, and that requires very fine discretization, and that's why it's very expensive to do, uh, the simulations in a classical way.
140
00:25:23.972 --> 00:25:34.212
And instead, if you want to move away from that and say we want to learn the features, which is what deep learning is all about, uh, then we cannot force it to be only in the Fourier domain.
141
00:25:34.272 --> 00:25:41.832
We have to give it nonlinearity to figure out what the right basis for it or the best basis to represent the signals are.
142
00:25:42.352 --> 00:25:56.652
And so that's the kind of like nice kind of combination, uh, we have that it's like all these nonlinearities will help it kinda, you know, find the right latent space. [laughing] Yeah. [laughing] No pun intended.
143
00:25:56.712 --> 00:26:07.752
That's- This is the right latent space. [laughing] It is the right... [laughing] And so, you know, the, the f- and if you do Fourier in that latent space, you know, that may be a more efficient way to represent.
144
00:26:07.812 --> 00:26:18.772
So that's one way of thinking because, you know, first of all, we are lifting the signal to more dimensions. Even if the signal is two or three dimensions, uh, we are now lifting it to much higher dimension.
145
00:26:19.232 --> 00:26:26.752
So in that space, the idea is it's easier to learn. Mm. And we're doing it as a nonlinear lifting, right? So there's already a latent space there.
146
00:26:27.132 --> 00:26:40.672
And then we are doing further nonlinear transformations in between our Fourier transforms. So that means we are saying yes, you know, maybe with these limited number of frequency modes, it's not expressive enough.
147
00:26:40.772 --> 00:26:53.392
But when I add nonlinearities, I can, you know, I can kinda nice, more nicely capture them. Yeah. Yeah. So you started your career back before neural networks were, I guess, taking off, right?
148
00:26:53.792 --> 00:27:03.292
So I think back then, people really did think a lot about, you know, appropriate basis sets and, you know, function expansions and orthogonal polynomials or whatever.
149
00:27:03.792 --> 00:27:18.752
How does that evolution from, you know, your research standpoint, how, like, as the community has evolved from that to, "Oh, just screw it, throw it all in," it seems like you still, like, believe in at least some of those concepts, um, as being guiding principles.
150
00:27:19.212 --> 00:27:35.112
Do you think that, that, that there is actually still lessons to be taken from, you know, classical, uh, mathematical, like rigorous mathematical techniques, um, that you can use those techniques to actually help improve modeling of the real world, uh, even if you still are just throwing the kitchen sink at things?
151
00:27:35.912 --> 00:27:47.352
No, I think it's a, it's a nice-- I think there's a trade-off involved. I mean, it's funny, my undergraduate thesis more than two decades ago now, uh, was on fractional Fourier transform, right?
152
00:27:47.452 --> 00:27:56.612
So and, and so yes, I mean, by themselves, like, you know, that wasn't enough to do computer vision, but I was curious, okay, what are these techniques and how well do they work?
153
00:27:57.232 --> 00:28:06.652
And so, you know, I'm completely with you that we cannot just force ourselves to use Stone Age techniques or [laughing] classical techniques. [laughing] Right?
154
00:28:06.732 --> 00:28:18.532
I mean, so we have to have feature learning, we have to have flexibility, expressivity. You know, they have to be easily optimized. So all of these aspects are very important with deep learning.
155
00:28:19.012 --> 00:28:32.812
But when it comes to the physical world and physical data, it's never going to be as plentiful as we see with language models because we are, you know, our weather model like had about like fifty thousand samples, right?
156
00:28:32.872 --> 00:28:40.812
Fifty thousand samples of fairly high resolution, like were global weather maps, but it's nothing like what we see with language.
157
00:28:40.892 --> 00:28:53.132
And, uh, in other domains it's even less because it's so expensive to simulate, and the real data may just not be available. Mm. And so here we have to think about the inductive biases more.
158
00:28:53.172 --> 00:29:07.212
We have to add in the physics constraints, it cannot be just reliant on data, and that's where I think a little bit more thinking of the architectural design comes up. Uh, the other aspect is computational complexity.
159
00:29:07.572 --> 00:29:16.072
So think about language. Uh, it's just one dimension, and even there the context length, you know, we are getting to millions and we are struggling, right?
160
00:29:16.172 --> 00:29:40.592
I mean, on the other hand, now we are thinking about, uh, not just two D, three D, even four D, you know, three D and time, and if each of the dimension is even a few hundred grid points, which is where, you know, industrial scale starts at like, um, like a thousand grid points in each dimension, we are talking like hundreds of billions to even a trillion context length, right?
161
00:29:40.652 --> 00:29:47.212
So forget ever having a transformer for anything of this scale. All of the world's compute will not be enough.
162
00:29:47.252 --> 00:29:55.652
And first of all, they all have to be co-located to [laughs] be w- able to ever do this, so that's why we need other architectures. But I would push back a little bit, right?
163
00:29:55.792 --> 00:30:05.572
We have the vision and video language models, right? And they use-- They basically learn a mapping- But the resolution is very low. Mm. That's the key.
164
00:30:05.632 --> 00:30:12.912
Like, for the physical world, the resolution, what do I req- we require, I mentioned like thousand by thou- Yeah... and by thousand by thousand. Mm.
165
00:30:12.952 --> 00:30:25.432
So there, you know, the, if you count that, that's like already in hundreds of mill- billions. Yeah. So, so you know, we are not doing that high resolution when we think about images and videos currently.
166
00:30:25.732 --> 00:30:33.732
I have a friend here- And, and the video is also like autoregressive, so it's essentially only like you only need to do the next step. Right. Yeah.
167
00:30:34.132 --> 00:30:37.792
But you're learning, I mean, like generally you're learning a code book, right?
168
00:30:37.892 --> 00:30:56.972
So you have, you're kind of learning the bias of the latent space or the, the real world to the latent space and, and, and so that if there is a, like a compression that you can do from the physical world into the latent space, then, then, um, you know, these autoregressive techniques have been successful.
169
00:30:57.112 --> 00:31:08.572
Yeah, but the idea is, you know, a lot of these autoregressive and, you know, techniques for, uh, vision and video models are for mostly like, you know, looking good, right?
170
00:31:08.652 --> 00:31:17.992
So they are like not for very precise simulations and they're, uh, you know, be- having that higher resolution I- and details is really important.
171
00:31:18.512 --> 00:31:25.792
And so we need to at least take in the data of that high resolution, right? So we need to be able to process that and reason over them.
172
00:31:26.232 --> 00:31:48.232
And so this is where a lot of the bottleneck is because we, you know, cannot afford to just throw away everything and say, "Oh, let's just like have hundred grid points in each dimension or fifty grid points," because there just isn't enough detail to correctly model, uh, phenomena like fluid dynamics, plasma, how materials deform.
173
00:31:48.312 --> 00:31:56.416
So all of this requires high fidelity, and for that we need high resolution. My understanding, you have a thesis that AI needs,
174
00:31:57.396 --> 00:32:07.116
you know, to incorporate the physical world into it in order to scale and be accurate going forward. Many people have this thesis.
175
00:32:07.776 --> 00:32:21.096
You are somewhat unique in that you have several examples of applying neural operators to the physical world in this way, and then it seems like you're constructing a thesis around your experience here.
176
00:32:21.156 --> 00:32:33.176
So can you share with us some of the really interesting, um, and exciting looking things that you've done using neural operators and other techniques like this? Mm-hmm. Yeah.
177
00:32:33.236 --> 00:32:43.876
I mean, you know, for us, when we started with like neural operators for partial differential equations, but also more broadly, you don't even need to assume they're partial differential equations, right?
178
00:32:43.916 --> 00:32:47.176
It could be any spatiotemporal or data at multiple scales.
179
00:32:47.696 --> 00:33:05.056
So we, you know, set out looking for interesting examples, and one of them was like weather modeling because the weather data is open source, uh, it's, uh, available, uh, called ERA5 from the ECMWF, the European, uh, Agency for Global Weather Modeling.
180
00:33:05.496 --> 00:33:13.456
And so given that the data was there, we were like, "Okay, let's just go try it," right? Yeah. And that's the beauty of it. Whenever data is available, uh, it's really good news.
181
00:33:13.556 --> 00:33:20.096
But a lot of weather scientists did caution us back then. This was back in '21, and they said, "No, no, no, this is so difficult."
182
00:33:20.416 --> 00:33:29.296
You know, there have been like decades of like development in traditional weather forecasting, and that's very careful bottom-up physics-based modeling, right?
183
00:33:29.356 --> 00:33:36.116
So assuming, oh, this is the fluid dynamics, can you go predict the weather, uh, the next day and so on.
184
00:33:36.436 --> 00:33:46.456
And so that's how a lot of the thinking was that AI is just not gonna be able to beat, uh, the, you know, decades of work in weather modeling.
185
00:33:46.936 --> 00:33:55.396
Uh, but to our surprise, we just went ahead, we trained them, we used neural operators to be able to effectively capture the phenomena.
186
00:33:55.796 --> 00:34:11.456
And then now we, uh, to our surprise, we found that it's not only, you know, accurate, it's almost as close to the, uh, what the traditional weather models can do accurately, but also tens of thousands of times faster.
187
00:34:12.036 --> 00:34:24.776
So what would take a big supercomputer to run can now be run, and we only needed a consumer-grade like GPU. Like, you know, it was a small model, it fit very well, it's very fast, and it's accurate.
188
00:34:25.096 --> 00:34:33.476
And I think that just changed everybody's thinking. And after that, DeepMind, Huawei, many others followed us a year later, released their own models.
189
00:34:33.576 --> 00:34:45.876
We were the first to actually open source our weather model, FourCastNet, and do it permissively. So that's what allowed companies, um, weather agencies, everybody to build on us.
190
00:34:46.036 --> 00:35:09.396
And so, you know, it's been a really interesting revolution to see that the weather models are now out there, uh, weather agencies are adopting them, and it allows us to now have small weather agencies in the, uh, Global South, for instance, have the same kind of fidelity that very big agencies were in the past only able to do, right?
191
00:35:09.436 --> 00:35:11.436
So it's democratizing weather modeling.
192
00:35:11.876 --> 00:35:24.496
And so that's just one example of where there's been very quick, rapid progress and a paradigm shift in terms of saying that, "Oh, now we can have AI as a reliable way to do weather modeling."
193
00:35:24.976 --> 00:35:38.556
I saw that, you know, the, the models that you mentioned, you made an insight that nobody else had, and that people were able to devise other mechanisms to kind of follow behind you. But there was some sort of,
194
00:35:39.456 --> 00:35:42.036
uh, shift in thinking that was required here.
195
00:35:42.416 --> 00:35:58.156
And, um, was it, it, was it simply, um, we believe that there's enough structure in this data to learn and that people are just doing it wrong, and people found other ways to learn the structure, but that your method was very, um- So let me, let me clarify, right?
196
00:35:58.216 --> 00:35:58.736
Yeah. So there is...
197
00:35:58.996 --> 00:36:18.376
You know, first of all, the very first work was to just say that, you know, look, traditionally this has been done with, uh, so trying to solve partial differential equations, each time doing it again and again, whereas AI learns from data, learns patterns, and can be just as accurate but fast.
198
00:36:18.516 --> 00:36:23.416
Yeah. And then the next iterations was to say, you know, how do we make it even more accurate?
199
00:36:23.836 --> 00:36:35.976
And there's the aspect that, you know, there is the short-term weather, like what is predictable for the next two weeks, and then there's the long term, you know, going to subseasonal to ultimately climate modeling.
200
00:36:36.576 --> 00:36:47.996
And, uh, traditionally what people did was to have different models for these different scenarios. So there's a different kind of system that works for short term, another system works for long term.
201
00:36:48.476 --> 00:36:57.856
But to me, there's only one Earth. You know, if you want a foundation model, if the claim is that it should be able to do both very short term as well as very long term together.
202
00:36:58.296 --> 00:37:09.176
And that's where in FourCastNet3, the latest iteration of the model, we're able to do both. And that's because you also, you know, incorporate the spherical geometry of the Earth.
203
00:37:09.616 --> 00:37:26.596
And so with a lot of the other architectures that have been, uh, getting fairly good accuracies for the short-term weather, when you l- run them for longer term, when you run them for like several months to even year, even before that, it just very quickly blows up, right?
204
00:37:26.656 --> 00:37:29.796
Because it assumes the world is a rectangle, which it isn't.
205
00:37:30.296 --> 00:37:44.396
And so incorporating all of the geometry and that, uh, information into neural operators means that we can faithfully run the same model also longer term and, you know, make this into a climate model.
206
00:37:44.456 --> 00:37:55.488
This is where the Allen AI Institute has now built climate models based on our neural operator architecture, and that's the only one that works- As an AI emulator, right?
207
00:37:55.548 --> 00:38:09.828
None of the other architectures work for climate because climate requires us to assume the world is a globe and that if you are repeatedly rolling out you kind of ke-keep that information.
208
00:38:10.308 --> 00:38:20.828
Uh, whereas if it's like a narrow surrogate, that's what I consider a weather model, you just narrowly look at a few metrics, many different architecture should do the job, right?
209
00:38:20.908 --> 00:38:38.608
But if you are asking one architecture do a range of different tasks like a foundation model, that's where incorporating the geometry of the Earth, which is that it's a sphere, and using neural operators, an efficient way to do that enables us to accomplish that.
210
00:38:39.268 --> 00:38:50.568
Yeah. So ForecastNet, um, you said trained on fifty thousand data points. Can we just talk about like what does this look like? Like, what does the data input look like? What are you actually trying to predict from here?
211
00:38:50.608 --> 00:38:58.388
And then what is sort of the, like the large scale, um, you know, you said you, you're going from weather to climate. What does it look like to do that generalization?
212
00:38:58.527 --> 00:39:08.668
Because I could imagine if you have fifty thousand data points, these are, you know, some high resolution in North America, then, you know, they might even depend on the local geography.
213
00:39:08.748 --> 00:39:20.228
Like, like if you are always modeling, you know, Kansas, is this gonna transfer to, um, you know, let's say the Swiss Alps or something, and then does that transfer to, uh, you know, the Himalayas?
214
00:39:20.888 --> 00:39:25.248
So first of all, to clarify, we're training it on the global weather model, right? Okay. Mm-hmm.
215
00:39:25.288 --> 00:39:35.528
So we have all the information around the Earth, and then we are, you know, s-asking it to predict, like, given the current weather, uh, like say wind conditions- Yeah...
216
00:39:35.628 --> 00:39:47.408
humidity and so on, what happens, uh, in a, in an autoregressive way, and it's every six hours. So what happens in the next six hours and so on, and you roll out, and you train the model to predict.
217
00:39:47.948 --> 00:39:55.888
And, and so, you know, you can have potentially the same model predict forever, right? [laughs] So- Yeah. And but the predictability window is like the weather. Mm-hmm. Yeah.
218
00:39:55.908 --> 00:40:09.168
And if you want to go beyond, you have to do what we call ensembles- Mm-hmm... meaning you have like a probabilistic estimate of what happens, uh, uh, in several months to years, and that's how you get a climate model.
219
00:40:09.468 --> 00:40:20.608
Ah, okay. So you have, you actually have an ensemble of these local predictors, and then you use the, some sort of statistics on the ensemble to get a climate prediction? Yeah. I see.
220
00:40:20.648 --> 00:40:30.168
So you kind of like have several rollouts. Essentially, you have several trajectories of rollouts, and then you get- And so the efficiency- You get probabilistic... is actually pretty key here. Yes.
221
00:40:30.228 --> 00:40:37.088
And that's why that's the biggest bottleneck with traditional climate modeling, that it's so expensive to do even one single run.
222
00:40:37.128 --> 00:40:50.188
You have to do long trajectories, very, very high resolution and, uh, you know, that's why we don't have a lot of very high resolution ability to do climate change predictions, for instance.
223
00:40:50.368 --> 00:40:59.128
So how do you validate climate? So you do lots of rollouts, and you look at sort of how things evolve kind of in aggregate. I mean, Nissi said butter-butterfly effect.
224
00:40:59.268 --> 00:41:12.148
Locally, I think we can believe even a perfect climate model or perfect weather model would only give you maybe two weeks before it sort of, it's becomes non-computable. So, um, you do lots of rollouts.
225
00:41:12.328 --> 00:41:25.028
Um, there's some chaos. You, uh, you average all these things. How do you actually validate that this works over long enough times? Yeah, timescales. Yeah. Yeah. And, and, you know, it's a tricky question, right?
226
00:41:25.108 --> 00:41:36.408
So for instance, you have to kind of ensure that you satisfy all of the physical constraints, and if you just do a standard rollout, that is, you know, likely not gonna happen. Mm-hmm.
227
00:41:36.428 --> 00:41:44.128
And so like some of the ongoing research we're doing is how do you kind of enforce the right physical constraints as you do the rollouts? Mm-hmm.
228
00:41:44.168 --> 00:41:53.888
Like, you don't want to, you know, completely wash out the fine details because then it's not accurate. Uh, but on the other hand, if you keep them, you may be physically they're invalid.
229
00:41:54.348 --> 00:42:09.348
So this is still an open problem, and that's what makes it difficult that you want AI to be fast, and you want to be able to do these, uh, long climate simulations and at the same time be able to have full confidence in them.
230
00:42:09.388 --> 00:42:20.968
But these are things we are working on now. I don't know if this falls into weather or climate, but we recently just had one of the most extreme heat waves in the history of modern climate data, right?
231
00:42:21.228 --> 00:42:28.188
Um, in the, just right here in the, uh, kind of southwest United States. I'm wondering, were you-- did you...
232
00:42:28.828 --> 00:42:39.048
I, I don't know if you were involved in this regularly, but do you know if you or anyone actually modeled that or predicted that correctly? Yeah. Yeah. So we have-- I know I don't have information on this specific one.
233
00:42:39.068 --> 00:42:50.688
Yeah. But we've, uh, tested in our latest, uh, ForecastNet three model extreme weather events of all kinds, right? Mm-hmm. And that's the key, like, you know, that's where you need probabilistic answers.
234
00:42:50.748 --> 00:42:57.628
So having just one deterministic output and saying that this is the weather is not enough when we are looking at extreme events. Mm-hmm.
235
00:42:57.648 --> 00:43:07.468
So we need careful probabilistic calibration, and we show that we are able to capture those well. And I think that was the surprise, even in our very first attempt- Mm-hmm...
236
00:43:07.508 --> 00:43:20.708
that we visualized certain hurricanes and storms, and it was able to do well, which is very surprising because you would think that rare events are not something AI would do well, right? Right.
237
00:43:20.708 --> 00:43:22.628
It would do well on typical events.
238
00:43:23.028 --> 00:43:35.968
But I think this is where more broadly the lesson is the physical world may be more forgiving because, you know, where there are extreme events like hurricanes that have very specific physical signature, right?
239
00:43:36.028 --> 00:43:43.628
So it's like extreme, but in a very specific way. So maybe you don't need as many samples because the physical world has a lot of structure.
240
00:43:43.988 --> 00:43:49.888
And that's something we see this again and again, that there is a lot of structure in so many other examples.
241
00:43:49.948 --> 00:44:08.168
You know, talk about plasma and fusion reactor- Uh, you know, we barely have a few thousand samples, but we are able to accurately predict, uh, events like disruption, uh, very well, and we are able to do that a million times faster than what traditional simulations were able to do.
242
00:44:08.668 --> 00:44:20.648
To me, yes, all these sound very surprising, but it's because I think the nature helps us a lot. You know, it has a lot of latent space structure [laughing] Yeah. That, uh- Structure, yeah.
243
00:44:20.788 --> 00:44:31.628
That I don't think traditional numerical methods are able to uncover because they are focusing more on correctness, that in any scenario you should be able to solve these equations.
244
00:44:32.068 --> 00:44:44.428
On the other hand, with AI, it's learning from data, it's uncovering the structure, it's uncovering how easy or s- kind of tractable these problems are, and that's what we are seeing in many cases. Yeah.
245
00:44:44.508 --> 00:44:55.908
I, I've had of a, a similar analogy. So from, you know, my own da- domain is probably closer to computational biology, but, you know, AlphaFold is the obvious, uh, like, really exciting development in the community.
246
00:44:56.508 --> 00:45:05.728
So the solving protein structure prediction and of course, all the caveats of what was actually solved, well, I think we discussed this in a, in a previous episode with, uh, the Boltz team.
247
00:45:06.188 --> 00:45:08.648
Encourage listeners to, to listen to that if they want more.
248
00:45:08.808 --> 00:45:26.048
But one of the, I think, key points about protein structure is that it really is constrained by physics, and that's why it was, in some sense, one of the, the few big wins in the field of biology, which is otherwise very complex, and we've had trouble, generally speaking, making a lot of success.
249
00:45:26.728 --> 00:45:39.608
And it seems like problems which are solved by differential equations or, I'd say, modeled well by differential equations, have a lot more, um, room for also integrating these techniques, um, as a general form. Yeah.
250
00:45:39.628 --> 00:45:43.088
Absolutely. I don't think there was a question there. Yeah. Absolutely. Yeah, editorializing. Yeah.
251
00:45:43.128 --> 00:45:58.108
So, so, so getting back to my question, um, we have climate or weather and climate, we have plasma, we have, um, uh, biology. Y- and I know that y- you, you prepared for us a few visualizations. Oh, yeah.
252
00:45:58.328 --> 00:46:07.208
Can you, can you just share with us, um, what does that look like? You know, what are the, so the visualizations and then what is the thread that runs through here?
253
00:46:07.248 --> 00:46:16.968
And I think maybe the re- the listeners will already have a hint about that, but I'd be really excited to see that. Yeah. Yeah, I can certainly share some of them.
254
00:46:17.108 --> 00:46:25.788
Uh, I mean, this one is just kind of showing that, you know, we have world at different scales, right?
255
00:46:25.848 --> 00:46:39.128
And these are examples of phenomena happening at different scales, from atomic to protein to even planetary scales, like the weather we talked about. And, you know, we need to capture all of that.
256
00:46:39.188 --> 00:46:47.448
That's what neural operators are, are designed to do, uh, and to kind of, you know, feed in data these different scales.
257
00:46:47.948 --> 00:46:55.708
And that's really, like, the aspect that makes a lot of the physical world problems hard, that need for fine scale.
258
00:46:55.768 --> 00:47:07.988
You know, we talked about how a lot of traditional computer vision video models are just designed to make things look visually good, and that requires low enough resolution.
259
00:47:08.048 --> 00:47:21.028
It's tractable, autoregressive, it's enough that, you know, it works out, it's short enough videos. But that's not how a lot of the physical simulation for, you know, industrial scale, high fidelity work.
260
00:47:21.168 --> 00:47:26.868
You really need high resolution. The, you know, ca- the atmosphere is one example.
261
00:47:26.948 --> 00:47:36.368
Like, you, depending on the resolution you observe, different phenomena can be captured, so you just miss that out if you don't have that high resolution.
262
00:47:36.708 --> 00:47:42.328
And now the question is, with AI, can we do this much faster than what we could with traditional simulation?
263
00:47:42.928 --> 00:47:56.808
And this one with neural operators is kind of showing that, you know, if you use the standard neural network and you had a fixed number of pixels like you're seeing on this side, and you zoom in, it gets blurry, right?
264
00:47:56.868 --> 00:48:02.648
That's the end of it. There's nothing beyond those, uh, fixed resolution that you can capture.
265
00:48:03.088 --> 00:48:22.848
But the idea is with neural operators, because it's a function space representation, meaning you can keep zooming in, you can add it the relevant details either by giving it data at higher resolution or physical constraints at higher resolution, you can kind of bring that, uh, multi-scale phenomena together.
266
00:48:23.088 --> 00:48:32.608
So where, where are the physical constraints? I mean, I assume physical constraints here are, like, local simulating, um, you know, fluid equations or some sort of- Yeah... hydrodynamic. Could be, right?
267
00:48:32.708 --> 00:48:45.268
So it could be of any nature. Mm-hmm. The idea is now you can add, like, conservation laws- Mm-hmm... for instance, in an incompressible fluid. Mm-hmm. You can add, like, material deformation, like how things stretch.
268
00:48:45.688 --> 00:48:57.268
So, uh, or it can be a full partial differential equation. Mm-hmm. So that's also an interesting question we've been researching, how is the curriculum of different physics, like you mentioned. Mm-hmm.
269
00:48:57.668 --> 00:49:09.048
You know, some physics may be very hard to impose or add as a loss function. Mm-hmm. Others may be easier. So you also need to kind of, you know, think about what, what to impose, you know?
270
00:49:09.248 --> 00:49:18.508
So, and the intuition is here is that when I'm adding a physical constraint, I'm adding it to the loss function. Is that more or less what's happening? Yeah. Because that's what is tractable, you know? Yeah.
271
00:49:18.528 --> 00:49:21.208
Making it a hard constraint is- Right... not tractable. Right.
272
00:49:21.568 --> 00:49:32.500
Whereas adding it as a loss function and, of course, there's still the balancing of that loss with the data we have, so we have to, you know, uh, do that in an appropriate way. Right. Yeah.
273
00:49:32.640 --> 00:49:46.040
So as I mentioned, the, this is the example of the weather model where, uh, here we are showing how, uh, we are able to capture, like, atmospheric rivers, which is the phenomena we see here in California- Every winter, yeah...
274
00:49:46.080 --> 00:49:54.780
you know, resulting in large storms. I think we're, we have one expected later this week, so [laughs] We'll, we'll hold you to that. Yeah.
275
00:49:54.840 --> 00:50:02.200
[laughing] Uh, so the idea of, like, you know, why show this is this kind of global phenomena, right?
276
00:50:02.280 --> 00:50:16.800
These are, like, thousands of miles wide, so you really need non-local models that capture, uh, these very, uh, large span phenomena and do that accurately, and that's what our neural operators are able to do.
277
00:50:16.980 --> 00:50:26.480
So, so the training data for this is... You were talking this a bit before, but I'm still, like, wondering what is this l- this is weather satellites, or is there ground-based, uh, data?
278
00:50:26.540 --> 00:50:35.740
Is it some hybrid of the, the two? It's, it's kind of a combination of different- Mm-hmm... sources, so it's what we call reanalysis data. So this is historical weather data- Mm-hmm...
279
00:50:36.140 --> 00:50:47.320
that is, in a way, reanalyzed, meaning that the sat-raw satellite data is combined with essentially what the physics solvers- Mm-hmm... tell you together a-assimilated.
280
00:50:47.880 --> 00:50:52.220
And so this is, uh, made available by the weather agencies, and we can train on them.
281
00:50:52.740 --> 00:51:03.120
So you're saying that they take a low-resolution data set, which is compiling all of the world's data set, you know, all of our meteorological data we have across the world- Mm-hmm...
282
00:51:03.200 --> 00:51:12.520
uh, and then they do short time simulations using physics-based, you know, classical techniques to, to fill in the details. Yes.
283
00:51:12.620 --> 00:51:22.220
Uh, and you can do that over short time spans, but as you go farther, it breaks down very quickly. So you're amortizing that across all the...
284
00:51:22.640 --> 00:51:30.200
Everyone would have to do that, and so somebody does it, and then you are able to take advantage. Yeah. I mean, this is data, right, that's already prepared.
285
00:51:30.260 --> 00:51:37.700
But the idea is already this data assimilation with physics kinda makes our model physics informed implicitly.
286
00:51:37.780 --> 00:51:47.420
So it's able to kinda, you know, k-keep that information, and that's why maybe that's one reason maybe it does well on even extreme weather events.
287
00:51:48.060 --> 00:51:58.440
Yeah, so this is just showing that we, our model is available in ECMWF, which is the weather agency, like, you know, the European weather agency.
288
00:51:58.560 --> 00:52:06.380
Uh, and so this was launched, like, more than two years ago, but, you know, I think fall twenty twenty-three. So, you know, s-
289
00:52:07.380 --> 00:52:21.220
I think ECMWF making these AI-based weather models available to the public, to me, was a very big step because that's where, you know, everybody could see what's happening. There were several hurricanes.
290
00:52:21.280 --> 00:52:28.060
Like, for instance, there was Hurricane Lee, and that's where the public could see what are these weather models doing.
291
00:52:28.160 --> 00:52:40.780
For instance, our ForecastNet was able to correctly predict that the hurricane m-making the landfall several days earlier compared to the standard, uh, weather forecasting models.
292
00:52:41.200 --> 00:52:53.680
And so the idea that these models could be very good for extreme weather events, uh, and do early prediction, you know, both for human lives, for economic costs, is a very big deal.
293
00:52:53.740 --> 00:53:03.120
And so that's when the public kind of got a lot more, I think, buy-in, uh, and from weather scientists because of how well it was doing in these events.
294
00:53:03.700 --> 00:53:14.500
And this is what I was talking about in a ensemble prediction, both for extreme weather or if you're thinking about climate. It's not just about looking at one trajectory, right?
295
00:53:14.580 --> 00:53:23.640
Because, you know, for unless you're somebody with a Sharpie somehow saying where the [laughing] hurricane is gonna go. [laughing] No, pun intended.
296
00:53:24.220 --> 00:53:41.240
But, you know, what you really want is the probabilistic prediction, meaning, you know, I'm gonna try different adding noise levels to my initial condition, what the, when the weather, uh, when the hurricane is forming in the Caribbean, I'm gonna add some noise because anyway, it's noisy.
297
00:53:41.280 --> 00:53:45.040
I don't know truly what the measurement there is.
298
00:53:45.420 --> 00:53:56.940
And then I'm gonna look at what happens to the possible hurricane tracks, and then I can come up with a probability of landfall in different regions, and that's how I can do risk assessment.
299
00:53:57.440 --> 00:54:04.000
And so this is where it gets even more expensive for traditional weather models because you have to run all of these ensembles.
300
00:54:04.340 --> 00:54:17.500
And now AI weather models being so fast, tens of thousands of times faster, means we can now do very large ensembles, and this is a very big improvement in terms of what we can do for risk assessment.
301
00:54:18.000 --> 00:54:29.720
Have you gone through and done, let's say, looked over the historical, uh, hurricane maps and then tried to do ensemble predictions and calibrated how often your predictions are, like a code? Yeah. Yeah.
302
00:54:29.780 --> 00:54:41.680
So in ForecastNet- Mm-hmm... three paper, there are, you know, uh, we have metrics of, like, extreme weather events- Mm-hmm... and ensemble prediction. And in fact, we've trained the model to do good ensemble prediction.
303
00:54:42.080 --> 00:54:53.120
Mm-hmm. And so this is where the calibration matters, uh, for, for these kind of events. Yeah. What was the sort of key insights or developments in ForecastNet three in versus two versus the first version? Yeah.
304
00:54:53.180 --> 00:55:02.060
So the first version was kind of the, you know, the using, like, the Fourier neural operators, but we didn't incorporate the spherical geometry, right?
305
00:55:02.140 --> 00:55:09.380
In this next version, we said, "I think, you know, it's important that the world is a sphere," because first of all, otherwise it's distorted- Mm-hmm...
306
00:55:09.420 --> 00:55:17.180
so you kind of are not predicting, uh, it in a- Sorry, weird dumb question, but if it wasn't spherical, what did you do? Like a Mercator projection or something?
307
00:55:17.440 --> 00:55:23.560
Yeah, the standard, like, kind of the, you know, like as you-- I, all, and all the other weather models do the same. Oh, really? Okay. Right?
308
00:55:23.620 --> 00:55:31.700
So they just kinda have the standard projection and then, uh, you know, predict the weather, and which is okay for short-term prediction.
309
00:55:31.760 --> 00:55:42.780
But when we, you know, our goal was to have the same model also do longer term, and that's when incorporating the spherical geometry added this additional stability. We could do longer roll-outs.
310
00:55:43.160 --> 00:55:48.540
And then in ForecastNet 3, the idea was it's not just about deterministic prediction.
311
00:55:48.860 --> 00:55:59.420
We want to get ensemble predictions right, so we have to train them based on this objective that we get the probabilistic predictions correct as well. How long are you predicting out?
312
00:55:59.940 --> 00:56:11.480
Um, and how many roll-outs are you doing? Yeah, so the... I know rollout is how long you predict, right? So each step is six hours, and then you predict for however long you want.
313
00:56:11.480 --> 00:56:23.420
[laughs] You know, you just have to roll out. Sorry, how many, how many, uh, examples in the ensemble do you have? So, a-and again, that's our choice. We can have- Yeah... like ensembles of different levels.
314
00:56:23.480 --> 00:56:34.420
So we-- I think it's like a few tens or something like is what we are currently, you know, shown, but you can do much larger too. And that's adequate to get out how far?
315
00:56:35.100 --> 00:56:44.180
My intuition is, the longer you want to predict, the more, uh- So not necessarily. Okay. It's really like about, again, calibrating the ensembles and ensuring- Mm-hmm...
316
00:56:44.220 --> 00:56:58.740
that they have the right spread rather than- I see... you know, so, so- Okay. So may-m-- So you have tens of these, um, models or examples in your ensemble, and that even with a very, very long roll-out, that's adequate.
317
00:56:59.120 --> 00:57:05.260
So again, like, you know, there's, as I said, a lot of still outstanding questions to do very, very long roll-outs. Mm-hmm. Right?
318
00:57:05.300 --> 00:57:14.340
Because you, you do need to incorporate like the physical constraints in a way to ensure that. That's something that we are actively researching now.
319
00:57:14.840 --> 00:57:26.480
Uh, but, uh, these models that we have are able to do the longest roll-outs compared to any of the other weather models that completely ignore spherical assumption and a range of other things.
320
00:57:26.980 --> 00:57:34.040
So when you say incor-incorporate, um, physical laws for climate over long times, I mean, what does that look like?
321
00:57:34.080 --> 00:57:43.180
Because there's a lot of, like, local conservation, which may be just broken if you take an ensemble of average, even though any given snapshot is, respects that.
322
00:57:43.220 --> 00:57:49.020
Or- No, the idea is to make sure you, uh, look at each ensemble- Mm-hmm... member and it respecting the physics. Okay, okay.
323
00:57:49.340 --> 00:57:58.400
You're not assuming the ensemble, uh, you're not deriving a, like, coarse-grained, like, equivalent of a probability- No... or something. Okay. Because then you would lose that- Yeah...
324
00:57:58.500 --> 00:58:07.520
you know, uh, resolution and the fidelity. Okay. That's a little confusing to me. Wait, so you're... So it's not an average. Well, how are you combining the ensemble?
325
00:58:07.640 --> 00:58:17.320
No, you are doing the average, but you're predicting each one. Oh, you're predicting each one separately. Yeah. Okay. Yeah, yeah. So each one is independently satisfies these constraints. Yes. But the ensemble- Mm-hmm...
326
00:58:17.600 --> 00:58:28.240
does not. Yeah, which is- And, and so that's how you ensure physical validity. I see. Okay. Yeah. So when you go on a sphere, you operate in, do you use basically spherical harmonics or some sort of- Yes. Okay, yeah.
327
00:58:28.680 --> 00:58:37.580
You have a, uh- Yes... spherical basis for it. Exactly. Which is actually very natural with Fourier netflow. Exactly. It's probably much harsher [laughs] if you're doing- Yeah... um, other. Yeah. Exactly.
328
00:58:37.620 --> 00:58:48.460
So that's where the, you know, like the Fourier saves us. Yeah. [laughing] Can incorporate these geometries well- Mm-hmm... and, and so I think be very faithful to, you know, what the globe is.
329
00:58:48.460 --> 00:58:58.200
[laughs] Yeah, arguably, it's much more natural- Yeah... than a, than like a Mercator projection- Yeah... or whatever other. Yeah, which [laughs] you know- Yeah... is like Greenland becomes huge. [laughing] Right.
330
00:58:58.220 --> 00:59:00.500
Exactly. Yeah. So that is a different story.
331
00:59:00.600 --> 00:59:14.440
But, uh, but yeah, but I think th-this is where I think the aspect of, you know, more broadly incorporating more of geometry and information about the domain becomes a lot more important in, for the physical world, right?
332
00:59:14.500 --> 00:59:25.420
So this is me again emphasizing that we need to incorporate more of the structures because one is the data is limited, and the other is a lot of what we are asking is extrapolation, you know- Mm-hmm...
333
00:59:25.540 --> 00:59:36.020
to go beyond than what it's trained on. You know, we're just training it to predict the next six hours and maybe do a little bit of multi-step fine-tuning for autoregressive roll-outs, right?
334
00:59:36.100 --> 00:59:45.420
So we are not training it to do very long, like a climate simulation, because that's just too expensive. But we hope magically it works well.
335
00:59:45.560 --> 00:59:59.840
[laughing] And it cannot if you just say, "I'm just gonna put a standard transformer or whatever else there," and it won't work out. So we added more of the domain constraints like spherical geometry.
336
00:59:59.920 --> 01:00:10.240
We add in maybe more of the physics in certain ways, and that's where it becomes more interesting algorithmically as well. You know, there's, uh, more in model design here.
337
01:00:10.720 --> 01:00:21.980
So the timescale that you train on, how, how long is that? To predict for the next six hours. Oh, j-- so only six hours? Yeah. That's abou- And a little bit of multi-step fine-tuning like together, like- I see.
338
01:00:22.020 --> 01:00:28.880
Yeah, yeah, I said that. Okay. Understood. So which is very surprising. [laughs] Yeah, that is very surprising. I would have expected it was weeks or months or whatever. Yeah.
339
01:00:28.920 --> 01:00:40.960
No, and then it kind of just works well even for like now we are showing for several months that it's able to do that. The number of steps is on- Hundreds or thousands.
340
01:00:41.000 --> 01:00:53.760
Yeah, and, and that your, your Fourier basis, you-- it's, it's many times the, the, the sort of base harmonic. I mean, this is like, that's in space, right? Yeah. So we're talking roll-out is autoregressive with time.
341
01:00:54.100 --> 01:01:06.560
No, but like in, in time, I'm, maybe I'm misunderstanding here, because it's a, in Fourier domain, right? No, no, in time it's not. That's what I'm saying. It's autoregressive in time. Oh, oh, I understand. Okay.
342
01:01:06.940 --> 01:01:12.740
So it's space, it is. Yeah, yeah. Yeah. Interesting. Yeah. Okay. What's the, uh, angular resolution- At least in, in this scenario.
343
01:01:12.780 --> 01:01:23.920
In other cases, we also have in-- time is also represented in the Fourier domain, and that's a question as well, can we do that? Right. But in this example, it's autoregressive. Got it. Yeah.
344
01:01:24.000 --> 01:01:30.940
Um, what's the angular resolution that you use for, for this, um, in, in, you know, in the spherical version? Yeah.
345
01:01:31.000 --> 01:01:44.480
So it's, so all of the data that's available is like, uh, I think a quarter, like point two five degrees. So that's- In terms of L maybe, or s-- I mean, in terms of like a spherical harmonic frequency- Uh...
346
01:01:44.500 --> 01:01:49.740
or, and I'm just curious. So this is like, uh-- Oh, you mean like how many modes we utilize? Yeah, I guess.
347
01:01:49.800 --> 01:01:58.140
I think we, so for that resolution, and we essentially utilize, I think most of it, only a few of them will leave out. I forget the details, but...
348
01:01:58.560 --> 01:02:06.880
I'm just curious, like what is the actual angular resolution on the globe that you are resolving to? Or maybe just the a- the, the physical resolution. Yeah. I mean, and that's what, uh- Yeah...
349
01:02:06.920 --> 01:02:17.400
that, because that's the data we get, right? So right now the data we get is like a quarter, like, uh, so point two five degrees. Totally. Oh, okay, like, uh, point two five solid angle.
350
01:02:18.068 --> 01:02:29.368
I think kind of comes out to like, uh, I don't know, seven hundred by few thousand, like, guide resolution. Mm-hmm. So but this is already standard, like, kind of processed. Yeah, yeah.
351
01:02:29.748 --> 01:02:36.268
And just, uh, trying to understand, like, how large of the basis do you need to represent this? Yeah, I mean, that's really depends on the resolution.
352
01:02:36.308 --> 01:02:48.128
And, and the idea is, you know, right now our weather data is just limited by this resolution. But if you could, you know, you could, like, kind of do synthetic climate simulations of even higher resolution, right?
353
01:02:48.188 --> 01:02:58.588
And that's kind of the next thing on how to combine these together. Do you think you can predict with super resolution be-- and basically resolution lower than the data provided?
354
01:02:58.908 --> 01:03:09.428
Again, like, yes, we can always predict them with- Sure... the neural operators. Uh, but you know, you do wanna incorporate more of the physical constraints to ensure that they are valid.
355
01:03:09.828 --> 01:03:15.688
Okay, so at, um, can we talk about- Yeah... some of the other, uh- Yes, yes, I know... technologies? There's a lot.
356
01:03:16.328 --> 01:03:27.768
Uh, so this is just showing, like, how, you know, the, what I described that the on the left wh- where the world is being assumed, it's a rectangle, it blows up very quickly. Mm-hmm.
357
01:03:27.828 --> 01:03:37.408
And on the right, because we assume the world was a sphere, it kept rolling it out, and it kept being stable. So- I still see a little bit of a singularity there- Yes... at the right [laughing].
358
01:03:37.448 --> 01:03:44.268
And it's still like, uh, you know, so the idea is, yes, because it's a very long rollout- Sure. Mm-hmm... and we have no guardrails of physics. Mm-hmm.
359
01:03:44.308 --> 01:03:56.628
We are not, you know, kind of projecting it to the right physics, right? This is full extrapolation. But the idea is the sphere assumption stabilizes it to a much greater- Yeah... extent.
360
01:03:56.648 --> 01:04:03.268
Compared to the left, it's, uh, much better. Yeah. Yeah. [laughs] But if you're, you're in the South Pole, you're still not gonna get exactly that. Exactly. So the poles are the hard part, so.
361
01:04:04.108 --> 01:04:19.768
So this is an example of the fusion reactor. So this is a tokamak, and we are able to model the complex, uh, plasma evolution and do this a million times faster than what we could do with traditional simulations.
362
01:04:19.988 --> 01:04:26.068
And, uh, this was the, in a way, we're creating a digital twin of the plasma, right?
363
01:04:26.147 --> 01:04:39.068
And then we can, you know, do further things like right now we are, as a next step, looking at, like, control, but with a full valid physics, like being able to prevent disruptions ideally and make fusion sustainable.
364
01:04:39.408 --> 01:04:45.428
So are you simulating MHD equations here or... Sorry. Yes. Magnetohydrodynamics equations? Yes, yes. Okay. Yeah.
365
01:04:45.448 --> 01:05:00.188
And then so for context, disruption in this case is this, uh, phenomenon which plagues, which plagues plasma physicists, where at some point, your entire plasma collects in a little tiny beam and then shoots a strong, you know, right through your containment vessel and blows it up.
366
01:05:00.308 --> 01:05:07.248
Yeah, and it can damage- Yeah... the reactor. Yeah. And that's the, that's a big bottleneck because then you have to kind of shut it down before that happens. Yeah.
367
01:05:07.268 --> 01:05:19.308
And then plus, it's no longer possible to have a sustainable fusion. Yeah. So there's a lot of open challenges here, but the idea is, you know, it's very expensive to go do physical experiments. Mm-hmm.
368
01:05:19.328 --> 01:05:31.368
The more you can capture that in a digital twin, but ensure physical validity, the more you can even do design and other considerations, uh, in the digital realm.
369
01:05:31.908 --> 01:05:36.028
You know, we can hopefully make advances, and these are the first steps towards that.
370
01:05:36.388 --> 01:05:50.468
The goal is that if you have one of these events, that you can somehow adjust the, the, um, the magnetic field so that it contains that and, and stabilizes. Yes. And that's the next step we are doing now.
371
01:05:50.528 --> 01:06:02.988
We are looking at, like, designing both the control and the simulation together. Nice. Are you working with a specific lab? I'm just curious. So this one was with the UK Atomic Energy Agency. Okay, okay.
372
01:06:03.028 --> 01:06:14.868
And now we're also working with a few others here in the US as well. So we are- Yeah... you know, kinda- Sure... getting the information from many different approaches of fusion itself. So this is the tokamak.
373
01:06:15.268 --> 01:06:22.228
We're also working with stellarators. We are working with different [laughs] approaches. The stellarators are- Yeah... are tricky. Yes. They're, yeah.
374
01:06:22.268 --> 01:06:32.908
But the idea is ideally, you know, like, our goal is to be able to design them in the digital twin. So can we come up with good designs that would make it maybe more practical?
375
01:06:33.428 --> 01:06:47.348
And so- And so that's, I think, also a good thing as an AI person and much more like, you know, agnostic and not picking a winner beforehand, right? Like, I like to work with different approaches, you know, uh, and see
376
01:06:48.388 --> 01:06:55.728
whether AI can accelerate all of them, and then we can kind of- Mm-hmm... you know, s- not prematurely rule out one approach over the other.
377
01:06:55.808 --> 01:07:07.288
So that's what AI enables us to be more kind of taking risks and exploring different approaches as opposed to in the physical world, trying to build any of these.
378
01:07:07.468 --> 01:07:14.208
You kind of have to cut a lot of the risky ones and say, "I'm only gonna do this because this is the most likely to work."
379
01:07:14.688 --> 01:07:21.688
I noticed over your career, you started out spending a lot of time on, you know, really theoretical foundations and mathematics of machine learning.
380
01:07:22.208 --> 01:07:32.868
And maybe, I don't know, something like six, eight years ago, you started working, really working a lot on applications and branching out in a diverse set of problems.
381
01:07:33.428 --> 01:07:44.488
What sort of prompted that shift in your, your, your approach and what you're looking at? I mean, so you're still working on very hard math problems as well, like, for example, the, the TorchLean work.
382
01:07:44.888 --> 01:07:54.128
But, um, the, the applications have really grown, and I'm wondering what, what prompted that? And, like, um, what were some of the lessons you've learned since then? Sure. Yeah. Sure.
383
01:07:54.188 --> 01:07:58.828
I mean, to me, it's like, you know, I feel like I've grown along with AI, right?
384
01:07:58.908 --> 01:08:18.508
So when AI was, you know, in this, where neural nets were not working because there wasn't enough data and all kinds of other reasons, you know, then you kind of have to build the theoretical foundations and try to hope that that leads you to a place where, you know, you get algorithms to work, right?
385
01:08:18.568 --> 01:08:26.728
And, and, and, and, you know, back then, like tensor methods was with that idea that, uh, you know, pre-deep learning, we still want a structure.
386
01:08:26.768 --> 01:08:38.136
We have probabilistic models like latent Dirichlet allocation for topic modeling and- Solving those were hard, but now tensor methods gave us a way to be very practical.
387
01:08:38.216 --> 01:08:43.096
It's parallel and can be done at large scale, but still has nice theoretical basis.
388
01:08:43.156 --> 01:08:57.936
So that was where, you know, was starting off, and then as deep learning started taking off and we could see that it works well in practice, and yes, there is a little bit of maybe theoretical understanding, but not a whole lot because of the way how complex it is.
389
01:08:57.996 --> 01:09:13.676
To me, theory should not be a constraint, right? It should be an enabler. And so that's where a lot of, like, the exploration was all to make this work well in practice and over into Amazon Web Services, then NVIDIA.
390
01:09:14.076 --> 01:09:23.196
So really like making things work at scale and really kind of getting hands dirty, right, was kind of like where a lot of the development is.
391
01:09:23.256 --> 01:09:35.256
And now I see a full circle because a lot of da-- purely data-driven approach is, in a way, seeing saturation, right? So now we want to ask, okay, either make them more hardware efficient, right?
392
01:09:35.336 --> 01:09:44.756
There's a lot of now room to kind of say, "Can we now, you know, make them much more energy efficient or hardware efficient?" So that's one aspect.
393
01:09:45.096 --> 01:09:57.416
But the other is areas like this where in the physical world, we don't have enough data. We are asking for hard extrapolation. You know, we want to think of doing discovery. By nature, it's about extrapolation.
394
01:09:57.476 --> 01:10:01.476
So we will never have data about a new discovery, right? That's by definition.
395
01:10:01.996 --> 01:10:15.236
And so there we need to again go back to thinking in principled ways, in the-- whether it's architecture design, algorithm design, the right loss functions. So we need to be much more mindful.
396
01:10:15.336 --> 01:10:27.236
So I see that coming a full circle because all of the things that work with deep learning, let's take them, but make them a bit more principled. There are several other applications which seem very natural.
397
01:10:27.636 --> 01:10:31.856
I'm wondering if you've worked on these or did I just miss some papers? If I did, I'm sorry.
398
01:10:32.396 --> 01:10:42.256
Um, so some examples are design of, like, electromagnetic circuits, um, I think is a big one, uh, or maybe not a big one, but I think will be coming up in the near future.
399
01:10:42.676 --> 01:10:54.576
Design of, let's say, materials, um, design of, let's say, dissipation and heat sinks, uh, or, um, or any sort of like fluid flow as I'm going through. What differential equations do I know?
400
01:10:54.616 --> 01:11:08.956
Electromagnetism, I know diffusion equations, um, you know, M-MHD. Yeah. Um, I'm wondering some of the other domains that are, um- Yeah, I mean, to me, there is just endless possibility, right? So there- Mm-hmm.
401
01:11:08.976 --> 01:11:15.436
You know, as like you can just have this work on any data and we have several other examples.
402
01:11:15.536 --> 01:11:25.556
So this was like, you know, being able to ask, can we sequester carbon dioxide underground and model, uh, how carbon dioxide, uh,
403
01:11:26.636 --> 01:11:35.996
expands or, you know, what is the pressure build-up in these reservoirs and, uh, you know, can we kind of model how they migrate over several decades?
404
01:11:36.476 --> 01:11:41.516
And so this one we were able to do much faster than what traditional simulations could do.
405
01:11:41.936 --> 01:11:50.096
I mean, the other aspect is being able to do all kinds of geometric shapes like, you know, being able to model aerodynamics in cars, planes and so on.
406
01:11:50.656 --> 01:12:03.956
And, uh, so again, this is a nice example of a latent space because you can transform a car or any other shape to a donut and then model on the donut and then transform the donut back to the car.
407
01:12:04.096 --> 01:12:09.416
So- You didn't turn it into a coffee cup. [laughing] Isn't that the classic joke? Your donut in your coffee cup. Exactly.
408
01:12:09.476 --> 01:12:23.996
So the idea of like a latent space to handle all kinds of different geometries and, and be able to capture the physics there in the latent space well means we can now have a model that generalizes across a lot of different geometries.
409
01:12:24.476 --> 01:12:39.356
Am I understanding that the, maybe the larger vision here is that, uh, you can train a foundation model in the sense of being able to model many different physical phenomena with the same model.
410
01:12:39.416 --> 01:12:56.436
And so you may fine-tune or there may be some kind of prompt that you give it to un-- have it understand the particular geometry, but that, that, that you, you know, sort of on all these different physical problems you train and then, and then you have your particular one and you're able to model that very effectively.
411
01:12:56.496 --> 01:13:03.216
Yeah. I mean, that's really the future, right? Because we have foundation models for language, maybe vision, but not for physics.
412
01:13:03.296 --> 01:13:12.176
So, you know, the idea is instead of like right now what we've seen are narrow surrogates, and we're trying to broaden their scope more and more.
413
01:13:12.336 --> 01:13:22.856
But ideally, we have much broader models that can work on a range of phenomena, but also multiphysics. So not just have like one single physics, but coupled physics.
414
01:13:23.116 --> 01:13:30.356
The real world has all of the physics kind of [chuckles] coming together in coupled ways. So can we bring all that together?
415
01:13:30.796 --> 01:13:40.336
So that's one aspect, like, you know, have foundation models that can do desi-- that can do simulation. But the other aspect that's really interesting is the inverse problem, right?
416
01:13:40.376 --> 01:13:44.976
So can I now not just simulate but ask what is the best design?
417
01:13:45.416 --> 01:14:06.176
And then these kinds of like, uh, models can like do simulation, but you can even do that implicitly and come up with the best design rather than in the earlier era of it was humans trying to come up with design, then you go and try to simulate or go to the wind tunnel, whatever physical testing and validate that.
418
01:14:06.216 --> 01:14:17.116
But now you have AI come up with optimized designs, but you have the guardrails of physics, so you have models that are accurate in physics, you have the confidence they work well.
419
01:14:17.576 --> 01:14:20.936
So you're kind of able to do that as well in the same model.
420
01:14:21.536 --> 01:14:37.336
Is there a reason-- Have you seen any evidence that you talked about these like sort of multiphysics being able to transfer or that, that, uh, you may be able to generalize to, um, sort of unseen physics? Mm-hmm.
421
01:14:37.996 --> 01:14:45.136
So, I mean, you know, like the physics by nature, if it's completely unseen, it's not possible to transfer, right? [laughs] Sure.
422
01:14:45.176 --> 01:14:51.176
I mean, I mean, if you're saying that we're going beyond the standard model- No, no, no, no, no... there's absolutely no data, that's not possible. Yeah.
423
01:14:51.216 --> 01:14:55.756
Uh, but if you're asking about like, you know, for instance, like I...
424
01:14:55.996 --> 01:15:10.156
You know, there is the like say I've like, you know, sh-shown it examples of like just how the heat propagates and the other examples of how the material like stretches and now there's coupling.
425
01:15:10.215 --> 01:15:21.336
Like, because of heat there's also stretching or kind of the joint phenomena. You could like now hope to fine-tune with much fewer samples because it kind of individually knows this phenomena.
426
01:15:21.856 --> 01:15:27.916
Then combining them together, maybe it can't do it from scratch because that's still too much to ask.
427
01:15:27.976 --> 01:15:38.916
It's highly nonlinear and coupled, but it can do it with fewer examples, and we've seen evidence of that in a lot of our papers that you're able to kind of essentially build up a curriculum.
428
01:15:38.996 --> 01:15:57.616
And that's what we see again and again in many of these examples that, you know, the real world we can kind of control a lot of curriculum and say, you know, let's kind of build in like modules and put them together, and that's what it now allows us to do in a systematic way here.
429
01:15:58.256 --> 01:16:03.256
I guess the design aspect, I don't know if we wanted to show, uh, very quickly.
430
01:16:03.336 --> 01:16:18.176
So this one was like, you know, looking at like designing the mask for inverse lithography, meaning now there's an inverse design problem, and we are also able to do that for designing gates and quantum dots.
431
01:16:18.296 --> 01:16:21.216
Uh, uh, this is like nonlinear photonics.
432
01:16:21.576 --> 01:16:40.696
And all of this, what is common is the idea that, you know, there's a forward model that is simulating the physics, but now what we want is the inverse design, like the problem that, uh, of we can optimize the best design, and humans are usually not good at this, right?
433
01:16:40.756 --> 01:16:51.656
[laughs] We are not good at like looking at highly nonlinear phenomena and say, "Oh, somehow maybe this combination of all these gates coming together helps pull the electrons together in a quantum gate."
434
01:16:52.076 --> 01:17:09.756
And so our collaborators were struggling to do that manually, and with AI we are now able to come up with very, uh, efficient designs, but also those we know actually work [laughs] because we have already the simulation as part of the loop saying that they work well.
435
01:17:09.816 --> 01:17:23.276
So I think these are examples where we see that it's not just about simulation, it's about really novel designs and novel discoveries that enable us to move the needle of innovation itself.
436
01:17:23.736 --> 01:17:26.876
Each one of these examples takes a lot of domain knowledge.
437
01:17:27.636 --> 01:17:41.276
How could somebody take your basic research if a domain expert and, uh, and quickly get started applying neural operators and the other frameworks that you've developed to their problem? Yeah.
438
01:17:41.776 --> 01:17:55.076
You know, neural operators are an open source library. It's extensively already adopted. It's part of the PyTorch ecosystem. It's, you know, used by a number of, uh, not only researchers, but also in companies.
439
01:17:55.396 --> 01:18:07.616
We have a lot of documentation there, so I encourage people to go there. We have like, you know, many different architectures, examples, recipes. So I think that's a great place to get started.
440
01:18:08.136 --> 01:18:17.776
You recently joined the UN Scientific Advisory Board. I know we're running out of time, but maybe just can you quickly give a bit of the story behind this and, uh, what you hope to accomplish? Yeah. Yeah.
441
01:18:18.176 --> 01:18:28.556
You know, I'm really honored to be part of the advisory board for the UN and in these tricky times with [laughs] a lot of geopolitics there, you know, which again, I'm not the expert on that.
442
01:18:28.616 --> 01:18:37.456
But when it comes to in-you know, aspects especially related to AI, having scientists in the room is something that, you know, I think is very important.
443
01:18:37.716 --> 01:18:52.196
I hope I could have an unbiased view and try to provide scientific evidence for any aspect, right? We wanna think about how AI impacts globally, like, you know, how do we ensure the benefits of AI reach everybody?
444
01:18:52.616 --> 01:19:04.576
How do we democratize access to AI? How do we ensure the unintended consequences and harmful impacts can be controlled? I think these are just the beginning aspects.
445
01:19:04.916 --> 01:19:15.056
Of course, the other side when it comes to weather models, I'm already excited, like, you know, there are, uh, there is a push to seeing how we can have better weather and climate modeling, so...
446
01:19:15.316 --> 01:19:19.096
then our food, you know, like using weather for better agriculture.
447
01:19:19.496 --> 01:19:31.116
So all these aspects are also where UN has a lot of dedicated agencies and people on the ground across the world, so I'm looking forward to contributing and being part of this.
448
01:19:31.636 --> 01:19:39.276
You know, looking, you know, looking at your career and how you, you know, talk and what you work on, it seems like you very much are a person who likes to solve concrete problems.
449
01:19:39.776 --> 01:19:46.696
You don't like to philosophize about things which... And you also seem, I think, maybe more optimistic than a lot of people in the AI space.
450
01:19:47.176 --> 01:20:02.616
Um, you have a very hope- like hopeful view of the world, um, I think not honestly always true. Um, what are the ways that you can uniquely bring that viewpoint to the board versus maybe some, you know- Yeah... living?
451
01:20:02.676 --> 01:20:08.136
Thank you. I, I... You know, to me, I think, as I said, I try to be unbiased and as a scientist- Mm-hmm...
452
01:20:08.236 --> 01:20:19.116
and, uh, as a scientist, I, I think that there's a lot of beneficial aspects of AI that are sometimes missed when we think of only the harmful impacts, right?
453
01:20:19.176 --> 01:20:36.636
And, and especially that is with respect to AI for science because a lot of regulatory frameworks equate AI with language models, and yes, language models can, you know, have manipulate people, can have all these kinds of harmful impacts that we should think about controlling.
454
01:20:37.136 --> 01:21:05.552
But AI for science is different, so I think this one size fits all is where a lot of- Problems come up, so we have to be mindful that there is, you know, AI that can change the world with new discoveries, and we should enable people around the world to not only benefit from them, but also be able to do research, you know, have access to AI that they can go innovate and use them in, uh, interesting ways.
455
01:21:06.132 --> 01:21:20.012
One question that we have been trying to ask every guest is, if you could pick a bottleneck in your domain that you could magically remove, what would that be and why? More compute. [laughing] Okay.
456
01:21:20.032 --> 01:21:24.592
[laughing] You know, that's the... I know that's an easy one or maybe a lazy one, right? Yeah.
457
01:21:24.612 --> 01:21:36.892
Because, you know, and, and you know, of course our compute that we have is growing so much more than even a few years ago, thanks to NVIDIA, thanks to others. Again- You may be biased there. Yeah.
458
01:21:36.912 --> 01:21:46.252
[laughing] So no, no comments on that. But what I mean by that is also like for research, enabling more and more compute, you know, it's very important.
459
01:21:46.392 --> 01:21:58.812
I know there are national labs building more com- supercomputers, uh, you know, hoping that we can have more compute for research. But I think, you know, without that we cannot experiment, we cannot innovate.
460
01:21:58.912 --> 01:22:06.672
I think this is a part that I push a lot and [laughs] you know, I think I cannot emphasize that it's so critical.
461
01:22:07.372 --> 01:22:17.752
If you had a call to action or a, um, something that you would like people to do or think about or, or learn about, what would that be?
462
01:22:18.352 --> 01:22:27.972
Yeah, so, you know, can go to Neural Operator Library, so you can kind of hands-on play with different architectures, recipes, you know, look at use cases.
463
01:22:28.512 --> 01:22:40.992
Uh, but also think about like, you know, AI for science as not just language models and agents. Yes, that's one aspect of it, but ultimately, you know, those are still like external wrappers in a way, right?
464
01:22:41.072 --> 01:22:54.732
Until we have AI that fully understands the physical world, not just as symbols, but as one that can simulate and design and control based on that, you know, there's a big piece missing.
465
01:22:54.832 --> 01:23:09.072
So that's the other aspect that I think the people should really think about AI for the physical world in this way. Anima, this has been so fascinating. Um, I, I'm excited to check out Neural Operators myself.
466
01:23:09.152 --> 01:23:17.532
I have some ideas in my head already. I really appreciate you taking the time to sit down with us. Thank you, RJ. It's been great. Thank you, Brandon. I really enjoyed it.
467
01:23:17.652 --> 01:23:31.392
We really dug deep into [laughs] a number of things. [laughs] So I appreciate you doing that. Thank you. Okay. Thank you. [outro jingle]
1
00:00:00.120 --> 00:00:09.180
So we, you know, set out looking for interesting examples, and one of them was like weather modeling because the weather data is open source.
2
00:00:09.320 --> 00:00:16.900
And so given that the data was there, we were like, "Okay, let's just go try it," right? Yeah. And that's the beauty of it whenever data is available, it's really good news.
3
00:00:17.020 --> 00:00:23.500
But a lot of weather scientists did caution us back then. This was back in '21, and they said, "No, no, no, this is so difficult."
4
00:00:23.880 --> 00:00:32.240
You know, there have been decades of development in traditional weather forecasting, and that's very careful, bottom-up, physics-based modeling, right?
5
00:00:32.280 --> 00:00:39.040
So assuming, oh, this is the fluid dynamics, can you go predict the weather, uh, the next day, and so on.
6
00:00:39.380 --> 00:00:48.640
And so that's how a lot of the thinking was that AI is just not gonna be able to beat, uh, the decades of work in weather modeling.
7
00:00:49.140 --> 00:01:08.760
Uh, but to our surprise, we just went ahead, we trained them, we used neural operators to be able to effectively capture the phenomena, and then we found that it's not only accurate, it's almost as close to what the traditional weather models can do accurately, but also tens of thousands of times faster.
8
00:01:09.320 --> 00:01:22.060
So what would take a big supercomputer to run can now be run, and we only needed a consumer-grade like GPU. Like, you know, it was a small model. It fit very well. It's very fast, and it's accurate.
9
00:01:22.380 --> 00:01:35.940
And I think that just changed everybody's thinking. Welcome to Latent Space. Um, this is the AI for Science section of Latent Space. Um, I'm Brandon. I work on RNA therapeutics using AI at Atomic AI.
10
00:01:36.320 --> 00:01:44.040
I'm joined by my co-host, RJ Haneke, who develops, uh, uh, spatial transcriptomics and is the CTO and founder of Mira Omics.
11
00:01:44.460 --> 00:01:52.580
Uh, today, we're excited to be joined by Anima Anandkumar, uh, the Brin Professor of, uh, Mathematics and Computer Science at Caltech.
12
00:01:52.940 --> 00:02:03.960
Um, Anima has done all sorts of really cool work, uh, combining AI with basically models of the physical world, um, and has a really diverse background. I don't think I could even remotely cover it.
13
00:02:04.520 --> 00:02:10.199
Um, but anyway, I'll let Anima introduce herself. Thank you for coming on the show. Yeah. Yeah. Thank you, Brandon and RJ.
14
00:02:10.240 --> 00:02:22.140
It's a pleasure to be there, and I really like the term latent space because that very much [chuckles] figures in a lot of my work, uh, because it's really, you know, the world is latent.
15
00:02:22.840 --> 00:02:37.600
Uh, but yeah, just as a brief introduction, uh, you know, I've been working in AI for more than two decades, uh, in a way, you know, before even deep learning when a lot of the theoretical foundations had to be built for probabilistic models, I worked on them.
16
00:02:37.980 --> 00:02:44.320
And then as deep learning started taking off, I also had a foot in industry until recently. So I was at NVIDIA.
17
00:02:44.420 --> 00:02:56.080
I led AI research there, and before that, at Amazon Web Services, helped found the cloud AI team and s- built the first cloud for AI products, uh, uh, back almost a decade ago.
18
00:02:56.500 --> 00:03:11.940
So, you know, like kind of having this one foot in industry and academia, I think has given me a lot of interesting perspective of how to bring theory and practice together and think of AI at large scale, but also AI that is principled.
19
00:03:12.440 --> 00:03:25.440
A lot of your work has been related to the modeling of physical systems, um, using s-certain types of physical systems, uh, which you model with differential equations, and you, uh, help model them with using machine learning.
20
00:03:25.540 --> 00:03:35.940
So, um, maybe first let's go in and talk a little bit about that as a high level, but we'll get to kind of the details about using neural operators and some of the applications like weather later.
21
00:03:36.460 --> 00:03:46.000
But first, I'm actually really curious to hear about TorchLean and how this, uh, recent work you've been doing connects with that, um, larger research program. Mm-hmm.
22
00:03:46.380 --> 00:03:53.300
To me, broadly, like, you know, my thesis is AI and science, how we bring that together, right?
23
00:03:53.460 --> 00:04:10.280
So, uh, you know, when I started at Caltech almost a decade ago, that's when, you know, my passion was always science, was physics, and but, you know, I was doing AI, so how to bring that together, uh, was where, you know, the first kind of foundations got laid there.
24
00:04:10.860 --> 00:04:22.820
And to me, like, you know, there are several aspects to that. One is people have been thinking how to use language models for science. Yes, you can do a lot of hypothesis generation.
25
00:04:22.900 --> 00:04:34.500
You can have ideas, but ideas are not enough, right? So you can have a lot of ideas. The bottleneck is going, testing, and verifying that they work in the real world.
26
00:04:34.940 --> 00:04:51.360
And so this aspect is where a lot of my recent focus has been on how do we ensure that we can build AI that has guarantees that it will work in the physical world or any aspects in scientific domains.
27
00:04:51.860 --> 00:05:07.420
And, and one way to think about it is, you know, can we model the physical world and keep the physics correct? And that's where neural operators come in. The other aspect is can we verify symbolically certain aspects?
28
00:05:07.580 --> 00:05:19.580
Uh, for instance, you know, if we, uh, claim that the theorem is correct, we have to go verify that. You know, that's where Lean as a formal language can be useful for verification.
29
00:05:19.960 --> 00:05:28.340
So how do we bring that together with language models is where a lot of mathematical reasoning has been at the forefront.
30
00:05:28.760 --> 00:05:43.300
And so TorchLean kind of is in that realm where we say, you know, not only that you wanna verify mathematical statements, you may wanna verify s- uh, what neural networks themselves claim to deliver.
31
00:05:43.700 --> 00:05:50.040
You know, for instance, if you're now using a neural network and you wanna ask whether it's gonna be robust.
32
00:05:50.240 --> 00:06:00.220
Say you wanna use a neural network in a control loop, or you wanna control, um, you know, whether it's a drone, whether it's, uh, a nuclear reactor.
33
00:06:00.540 --> 00:06:14.060
So all of this ultimately when we br- build AI systems with deep learning into control loops, we want robustness. And so now TorchLean can help us do those verifications seamlessly.
34
00:06:14.520 --> 00:06:22.320
So we can now have neural networks be part of the verification loop and have confidence that we can use them appropriately.
35
00:06:22.940 --> 00:06:36.088
We have already discussed on the podcast Lean and, uh, and ev-everyone should probably be familiar with neural networks. How- But neural networks seem very unconstrained. What kinds of proofs are you talking about?
36
00:06:36.148 --> 00:06:44.868
Are you talking about bounds on the outputs, inputs? Um, what, what can you prove with- Yeah... Torchlean? Yeah. So Torchlean is a, an overall framework, right?
37
00:06:44.948 --> 00:06:51.228
So what it really enables is that you can now write neural networks essentially in Lean.
38
00:06:51.728 --> 00:07:04.008
So instead of writing in like PyTorch, it's like a PyTorch-like abstraction, but you can like kinda, you know, write it in Lean, and so it can be fully formalized in Lean. And then there are several implementations.
39
00:07:04.068 --> 00:07:14.188
You know, we have algorithms for certified robustness like Crown. You know, those are implemented under this framework. So- Sorry, what was it? Like what? Like Crown? Crown.
40
00:07:14.368 --> 00:07:25.548
Crown is one of the, um- Can you explain a little bit? So there are different ways to bound, you know, for certified robustness. You know, how tight those bounds can be, it depends on the relaxation techniques.
41
00:07:26.028 --> 00:07:37.608
And sort of without going into those, there's many such algorithms. But, you know, we are kinda like implementing them and enabling them in Lean, so we can seamlessly run both.
42
00:07:37.868 --> 00:07:50.588
You know, we can both first kinda write down in a Torch-like framework neural networks very simply, right? And then we can also make statements about them formally and verify them.
43
00:07:50.688 --> 00:08:01.488
So all of that can be brought together in one framework. So what's an example of a, a bound that you could claim? Like, so we're operating a nuclear reactor, we don't want it to melt down.
44
00:08:01.768 --> 00:08:13.608
What are the sort of guarantees that, um, you could provide to the out, inputs and outputs that would help that not melt down? Yeah. I mean, the natural one is the certified robustness that I mentioned.
45
00:08:13.688 --> 00:08:25.388
So saying that if you-- our inputs are, you know, perturbed by a certain amount, how much is the output gonna be perturbed, right? The sensitivity analysis, uh, is another term.
46
00:08:25.748 --> 00:08:41.648
And so having those kinds of bounds for different neural architecture, so you kinda automatically get those bounds, can then help us, you know, not only train neural networks to do well in a control loop, but also worry about safety and robustness or stability.
47
00:08:41.768 --> 00:08:47.368
These are all part of control systems that people worry about, so that's one example of an application.
48
00:08:47.888 --> 00:08:58.148
So it's really more broadly the idea is you need verification, uh, in lots of scenarios that involve neural networks. So control loops are one.
49
00:08:58.548 --> 00:09:12.328
Another example is, you know, we used physics-informed neural networks to, say, solve partial differential equations or come up with systems that gua-- that are guaranteed to satisfy certain physical laws.
50
00:09:12.788 --> 00:09:24.868
But we also wanna verify, for instance, that our neural network is only trained in finite precision, right? So can we overcome those requirements, and what happens when we, uh, are...
51
00:09:25.348 --> 00:09:34.408
What are the shortcomings because we are using this finite precision? Can we also bound those? Oh, right. So those are other kinds of bounds that work in Torchlean.
52
00:09:34.468 --> 00:09:50.728
So all aspects of like, uh, you know, the effect of precision, the effect of perturbation, all of these we can, uh, you know, we can have algorithms that are implemented in Lean that can be seamlessly now part of the verification loop.
53
00:09:50.888 --> 00:10:00.468
So, and do-- is the descriptive power of the Torchlean, is that sufficient to describe basically any neural network, or is there, are there constraints on that? Yeah.
54
00:10:00.528 --> 00:10:10.588
So it's essentially a, you know, PyTorch-like, you know, framework, right? So you can just kinda nicely define neural net layers in the same way.
55
00:10:11.148 --> 00:10:23.388
Uh, but the back end having like Lean, uh, helps us formalize and prove it. And for like transformer architecture, for example, is it reasonable to prove these kinds of bounds on a very large neural network?
56
00:10:23.448 --> 00:10:34.508
So the- Or transformer. You know, there is the aspect of one is like kinda having the framework, right? The other is scalability. Lo-- So Lean still has a lot of, uh, shortcomings there.
57
00:10:34.588 --> 00:10:47.948
It's CPU-based, and, uh, you know, it's not, uh, like make getting that onto the GPU has a lot of- Yeah... nuances there. Uh, so, you know, a lot of work needs to be done. So what we've started with is a framework.
58
00:10:48.047 --> 00:10:57.188
You know, making that more efficient, especially at a very large scale, requires still a lot of work to be done. Yeah. But that's true broadly for Lean as well.
59
00:10:57.848 --> 00:11:00.568
And just, uh, trying to understand like how we imagine this.
60
00:11:00.608 --> 00:11:13.908
So if I were to take a, say, numerical analysis classes, you know, on graduate level numerical analysis class, you have a differential equation, you have some discretization error or something, and you bound like given these properties, I can bound the solution.
61
00:11:14.228 --> 00:11:24.028
Mm-hmm. Right? So solving, um, some of these physics-based, uh, or AI-based solutions to differential equations, um, I think historically has been kind of the wild west.
62
00:11:24.088 --> 00:11:30.168
So I think you mentioned physics-inspired neural networks. Um, really cool idea. It'd be fun to talk about that a little bit.
63
00:11:30.728 --> 00:11:38.968
But I, I know that sometimes they, they are particular, and that people don't-- they don't always work, and I think people don't always know when they will or won't work.
64
00:11:39.328 --> 00:11:51.828
Um, I, I mean, I'm not an expert, but I'm, I'm just wondering if that's, um, in your experience. Um, and what I'm wondering is like, has this helped you understand like w-- the domain of applicability for PINNs or...
65
00:11:52.368 --> 00:12:04.168
And is that sort of like the goal is like you can rigorously say like, "This solution will converge," or is there not necessarily the same concepts of convergence in a controlled way for neural networks? Yeah.
66
00:12:04.228 --> 00:12:18.288
So, you know, like physics-informed neural nets are about like saying that, you know, I write down like a PDE, uh, partial differential equations, and hopefully the optimization succeeds and I get the answer, right?
67
00:12:18.668 --> 00:12:28.368
And of course, if like optimization was not at all an issue, this would be universal. You solve everything, you know, we're all happy, but that's not the case.
68
00:12:28.748 --> 00:12:45.488
And so optimization ends up being usually very difficult, especially for problems that are time-dependent, meaning it's not just stationary, you also have time, and the time component in many cases could be turbulent, like in the case of fluid dynamics.
69
00:12:45.908 --> 00:12:57.224
You know, you kinda like if you- Run it long enough, uh, it can become chaotic, so you really, you know, have, like, very small fine-scale effects matter.
70
00:12:57.864 --> 00:13:06.344
And so in those cases, just trying to solve a partial differential equation at all times, uh, is just hopeless.
71
00:13:06.444 --> 00:13:13.504
Like, you know, this is not an optimization landscape that, uh, you know, I think we'll, you know, we can have any handle on.
72
00:13:13.884 --> 00:13:24.724
And this is where the idea that from scratch we would be able to solve these, uh, equations using a neural net is not possible, so PINNs don't work everywhere.
73
00:13:25.304 --> 00:13:38.624
And our idea of neural operators came as a way to overcome this, right? So saying, you know, we can't rely just on physics constraints alone to come up with answers. We have lots of data available.
74
00:13:38.864 --> 00:13:51.184
Um, you know, I'll talk about the weather example where we even collect data, right? So we don't just solve equations and g- have synthetic data, but we also have real data by observing the weather as one example.
75
00:13:51.584 --> 00:14:07.704
So why not make use of all of the data available so we don't just rely on trying to solve partial differential equations and other physical problems from scratch, because it's really the data-driven approach that makes it possible to get quick answers.
76
00:14:08.084 --> 00:14:21.244
And so with neural operators, we can bring both of them together. We can have all the data that's available. We can utilize it. We can add physical constraints, and then that overcomes the limitations that PINNs face.
77
00:14:21.844 --> 00:14:27.244
Can you give a little bit more intuition on the difference there and why that is possible?
78
00:14:27.324 --> 00:14:39.924
So I heard you mention, you know, in, with PINNs, you're basically just baking the physics constraints into the neural network, but that this becomes unstable over time or other, other variables.
79
00:14:40.444 --> 00:14:49.104
Whereas if you add a little bit of data, like I, I can kind of un- intuitively understand why that might help, but can you give a little intuition for what's going on? What's the difference here?
80
00:14:49.584 --> 00:14:57.604
So with the PINN, like, you know, every instance of an equation you solve from scratch, right? At least in the classical sense.
81
00:14:58.044 --> 00:15:08.064
So you start, you take s- the specification of what equation you want to solve, and you hope that the optimization landscape succeeds, which many cases it doesn't.
82
00:15:08.524 --> 00:15:20.144
Whereas with the neural operators, what we do is we, you know, have lots of tr- data, so we have a training phase. Uh, we teach it how to come up with solution for different instances of equation.
83
00:15:20.684 --> 00:15:29.824
And so just as in other supervised learning, at test time, you can now ask, you know, "Can you come up with an answer?" And you can still have physics constraints as a way to guide that.
84
00:15:30.404 --> 00:15:41.904
Uh, so, you know, it can be both data-driven and physics informed together. Uh, but the benefit is because we have data, you know, it's m- like you're not stuck in an optimization landscape, right?
85
00:15:41.984 --> 00:15:50.744
So you know what the answers are during training, so you're now at a better chance to come up with the right answers even at test time.
86
00:15:51.184 --> 00:16:05.864
My understanding is that a neural operator is, is, uh, a function fit to, to, uh, like, to data or a neural network, you know, learns to fit functions to data. Is that a good intuition here? Yeah.
87
00:16:05.924 --> 00:16:13.244
So, you know, neural operators are, in that sense, similar to... You know, it's the same as neural networks, right? You're learning on data.
88
00:16:13.604 --> 00:16:24.764
But the difference is neural operators are, you can think of it as a generalization of neural networks. So with standard neural networks, the inputs and outputs are of fixed sized.
89
00:16:25.204 --> 00:16:29.344
So in language, we have fixed vocabulary, we fix what the input and output are.
90
00:16:29.744 --> 00:16:41.544
And same with images, uh, in computer vision and videos, we assume a fixed resolution, and we always, you know, our inputs and outputs are always at that fixed resolution. We can't change it post-hoc.
91
00:16:42.104 --> 00:16:52.184
Whereas with a lot of this physical data, the idea is our world is inherently multi-scale, so you should not be, like, deciding beforehand what the resolution is.
92
00:16:52.624 --> 00:17:01.344
You know, maybe you have, like, weather data available only at coarse resolution, but really the actual phenomena is happening at a finer scale, right?
93
00:17:01.424 --> 00:17:10.864
And maybe you wanna after that incorporate, uh, either additional data at finer resolution or add in physical constraints at finer resolution.
94
00:17:11.304 --> 00:17:22.144
So we should be having that flexibility, and we should really think of the world not at these fixed resolution, but one that's happening, uh, infinitely [chuckles] you know, that at one...
95
00:17:22.624 --> 00:17:25.444
The real world happens at that infinite resolution.
96
00:17:25.864 --> 00:17:36.304
And that's what neural operators enable because they model inputs and outputs as continuous functions that can be infinitely resolved, that can have infinite discretization.
97
00:17:36.904 --> 00:17:45.804
And now we can have, uh, you know, at inference time, you can give it now inputs and, uh, ask for outputs at any resolution.
98
00:17:46.264 --> 00:17:59.144
So you're not just limited to the resolution of training that we see in standard neural networks, and that's what neural operators enable. So neural operators enable us to zoom in and out as we like.
99
00:17:59.744 --> 00:18:12.844
So obviously that, as stated, that any function that, that's under a constraint, right? You could have many, many functions that fit this, fit the data. It would be easy to overfit. So how do you regularize that? Yeah.
100
00:18:13.284 --> 00:18:25.844
Certainly, like, you know, uh, you know, if you're, uh, asking about making predictions at a higher resolution than what's seen, like what we call zero-shot super resolution, uh, you're kind of making some guesses, right?
101
00:18:25.924 --> 00:18:33.344
And, and that's what these models are doing. They're trying to regularize and kind of smoothly extend to higher resolution.
102
00:18:33.664 --> 00:18:51.164
But of course, if you now give it the model additional information in terms of, let's say, a physical loss, so you could give it pa- partial differential equation constraints, conservation laws, and you can now enforce them at a finer resolution than the data you have, then there's more guidance in a way.
103
00:18:51.644 --> 00:19:02.104
So that way it can now come up with the, uh, the right answers even at higher resolution because you're, you know, giving it constraints at higher resolution.
104
00:19:02.164 --> 00:19:13.224
And so that's how we can ensure that these physics informed neural operators can work at higher fidelity and higher resolution than even the training data that was available.
105
00:19:13.840 --> 00:19:30.180
My understanding is a, a lot of your work uses a particular kind of, uh, neural operator, a Fourier neural operator. So Fourier is a dual domain, it is extended across the entire, uh, domain of the inputs.
106
00:19:30.740 --> 00:19:37.960
That's a lot of jargon, maybe. Can you give some intuition for why, why is that important? How does that help?
107
00:19:38.500 --> 00:19:50.540
What I mentioned neural operators as a class of, uh, mo- models that allow us to have any resolution input and any resolution output, right? And learns the mapping between them.
108
00:19:50.820 --> 00:19:58.240
You know, that's really called an operator. So [chuckles] the mapping between function spaces. So that's the reasoning behind the name neural operator.
109
00:19:58.760 --> 00:20:07.100
And, uh, you know, Fourier neural operator was one of the early, uh, setups we y- or architectures we came up with.
110
00:20:07.540 --> 00:20:20.980
And the reason why that's been so successful is because it kind of strikes a nice trade-off between efficiency and expressivity, right? So why is the Fourier space a good one?
111
00:20:21.420 --> 00:20:29.400
The Fourier space allows us to, you know, it's a dual space, like you mentioned, but it really allows us to capture non-local phenomena, right?
112
00:20:29.520 --> 00:20:36.860
So meaning something that's like, uh, a non-local in the Fourier domain could be even efficiently captured.
113
00:20:37.400 --> 00:20:48.700
And there are a lot of phenomena like we see in nature, whether it's fluid dynamics, material deformation, quantum chemistry, it's all, you know, there's a lot of them are non-local.
114
00:20:48.940 --> 00:20:57.360
You know, the differential equation, like the derivative is local, but the inverse of it is you're kind of doing essentially integration, it's non-local, right?
115
00:20:57.420 --> 00:21:03.160
So the solutions are non-local, and these models are able to capture that.
116
00:21:03.480 --> 00:21:15.800
But at the same time, doing Fourier transform is efficient, and it kind of like nicely captures a lot of inductive bias we see in many of these, um, natural phenomena.
117
00:21:16.120 --> 00:21:28.880
But this doesn't mean that we are capturing the world entirely in the Fourier sp- uh, basis, right? It's not a linear representation in the Fourier basis, which is what classical numerical methods do.
118
00:21:29.340 --> 00:21:38.760
We add non-linearity just as in transformer and other neural nets in between Fourier layers. And we also add residual collections.
119
00:21:38.820 --> 00:21:51.080
So all of these architectural aspects that are inspired by other neural nets that work well in other neural nets, bringing that together really kind of helps us get best of both the worlds.
120
00:21:51.120 --> 00:22:03.440
So you can think of like if we were to use transformers and we require a very high resolution, it would become untenable because of the quadratic complexity and all to all connections.
121
00:22:03.880 --> 00:22:17.520
On the other hand, if you did that with Fourier transforms, uh, we have like quasi-linear, uh, complexity, and still we have global connections in a way we can model these non-local phenomena.
122
00:22:17.860 --> 00:22:24.720
And so that's why it's a nice middle ground. So that allows you to learn from what is happening on the other...
123
00:22:24.760 --> 00:22:33.820
Like, it- it's talking weather, like what's happening in Chicago may have some impact on what's happening in San Francisco, or maybe not, but that's the idea. Yeah.
124
00:22:33.860 --> 00:22:46.680
So that's the idea in time, like kind of, yes, at this part may be local, but eventually they have an impact in other locations and, uh, yeah, so both in space and time, we want to capture that dependence.
125
00:22:46.840 --> 00:22:57.900
Yeah, so what happens today in Chicago will happen-- will have an impact in a month in San Francisco or something like that. Uh, yeah. So w- you know, so there is like both the short-term and the long-term effects.
126
00:22:57.940 --> 00:23:04.800
So in a short-term, like we think about predictable weather, but longer-term, we're talking about climate, right? Right.
127
00:23:04.900 --> 00:23:18.860
So what happens, you may not be able to say precisely, you know, what happens in Chicago, what will happen in Ch-San Francisco. Yeah. That's like the butterfly effect. On the other hand, we can kind of give averages.
128
00:23:18.920 --> 00:23:30.360
You know, if there's heat wave in this kind of overall region, you know, we kind of have an idea that it's gonna be higher than average temperatures. So those are the aspects we can capture together.
129
00:23:30.900 --> 00:23:48.960
From an architectural standpoint, for all the AI engineers here, are we just talking about doing all the work in the Fourier domain, but it's basically the same neural network, but I'm just operating in the Fourier domain, or is there other, other aspects that are cons- that are required in order to do this properly?
130
00:23:49.040 --> 00:23:53.620
So think of it, I guess the, maybe the easiest way to think about it is, you know, you can...
131
00:23:53.960 --> 00:24:03.220
If you think of a transformer architecture instead of like the, uh, you know, attention map, you now have the Fourier, but you still have other non-linearities.
132
00:24:03.320 --> 00:24:11.360
You have like, you know, the residual, you have, you know, many other parts of the architecture still there that give it like expressivity.
133
00:24:11.720 --> 00:24:23.180
And we have lifting to higher dimension, like, you know, in a channel space to give it more expressivity. So all of those kind of best principles are, are still available. Mm-hmm.
134
00:24:23.260 --> 00:24:34.040
Uh, but the Fourier helps us capture that all to all, uh, you know, dependence without requiring very huge complexity that we see in transformers. That makes sense.
135
00:24:34.560 --> 00:24:39.540
The other advantage is that it gives you the natural multi-scale with sort of a implicit cutoff.
136
00:24:40.180 --> 00:24:53.080
Is, um, you know, the sort of, if you have a signals background or physics background, you might ask, uh, you know, in linear, if you're doing everything linearly, there's a maximum frequency and you know, you know, above that you can't represent anything.
137
00:24:53.720 --> 00:25:03.620
Um, but how does that-- how does adding these other architectural changes in a non-linear domain actually affect, you know, your choices of frequency bounds and- Yeah. Yeah.
138
00:25:03.660 --> 00:25:07.520
No, that, that's a great question, and that's where the expressivity comes in, right?
139
00:25:07.620 --> 00:25:23.380
Otherwise, if you're just taking a Fourier transform of a signal and trying to represent it, you know, that's what numerical methods have also attempted to do, and that requires very fine discretization, and that's why it's very expensive to do, uh, the simulations in a classical way.
140
00:25:23.972 --> 00:25:34.212
And instead, if you want to move away from that and say we want to learn the features, which is what deep learning is all about, uh, then we cannot force it to be only in the Fourier domain.
141
00:25:34.272 --> 00:25:41.832
We have to give it nonlinearity to figure out what the right basis for it or the best basis to represent the signals are.
142
00:25:42.352 --> 00:25:56.652
And so that's the kind of like nice kind of combination, uh, we have that it's like all these nonlinearities will help it kinda, you know, find the right latent space. [laughing] Yeah. [laughing] No pun intended.
143
00:25:56.712 --> 00:26:07.752
That's- This is the right latent space. [laughing] It is the right... [laughing] And so, you know, the, the f- and if you do Fourier in that latent space, you know, that may be a more efficient way to represent.
144
00:26:07.812 --> 00:26:18.772
So that's one way of thinking because, you know, first of all, we are lifting the signal to more dimensions. Even if the signal is two or three dimensions, uh, we are now lifting it to much higher dimension.
145
00:26:19.232 --> 00:26:26.752
So in that space, the idea is it's easier to learn. Mm. And we're doing it as a nonlinear lifting, right? So there's already a latent space there.
146
00:26:27.132 --> 00:26:40.672
And then we are doing further nonlinear transformations in between our Fourier transforms. So that means we are saying yes, you know, maybe with these limited number of frequency modes, it's not expressive enough.
147
00:26:40.772 --> 00:26:53.392
But when I add nonlinearities, I can, you know, I can kinda nice, more nicely capture them. Yeah. Yeah. So you started your career back before neural networks were, I guess, taking off, right?
148
00:26:53.792 --> 00:27:03.292
So I think back then, people really did think a lot about, you know, appropriate basis sets and, you know, function expansions and orthogonal polynomials or whatever.
149
00:27:03.792 --> 00:27:18.752
How does that evolution from, you know, your research standpoint, how, like, as the community has evolved from that to, "Oh, just screw it, throw it all in," it seems like you still, like, believe in at least some of those concepts, um, as being guiding principles.
150
00:27:19.212 --> 00:27:35.112
Do you think that, that, that there is actually still lessons to be taken from, you know, classical, uh, mathematical, like rigorous mathematical techniques, um, that you can use those techniques to actually help improve modeling of the real world, uh, even if you still are just throwing the kitchen sink at things?
151
00:27:35.912 --> 00:27:47.352
No, I think it's a, it's a nice-- I think there's a trade-off involved. I mean, it's funny, my undergraduate thesis more than two decades ago now, uh, was on fractional Fourier transform, right?
152
00:27:47.452 --> 00:27:56.612
So and, and so yes, I mean, by themselves, like, you know, that wasn't enough to do computer vision, but I was curious, okay, what are these techniques and how well do they work?
153
00:27:57.232 --> 00:28:06.652
And so, you know, I'm completely with you that we cannot just force ourselves to use Stone Age techniques or [laughing] classical techniques. [laughing] Right?
154
00:28:06.732 --> 00:28:18.532
I mean, so we have to have feature learning, we have to have flexibility, expressivity. You know, they have to be easily optimized. So all of these aspects are very important with deep learning.
155
00:28:19.012 --> 00:28:32.812
But when it comes to the physical world and physical data, it's never going to be as plentiful as we see with language models because we are, you know, our weather model like had about like fifty thousand samples, right?
156
00:28:32.872 --> 00:28:40.812
Fifty thousand samples of fairly high resolution, like were global weather maps, but it's nothing like what we see with language.
157
00:28:40.892 --> 00:28:53.132
And, uh, in other domains it's even less because it's so expensive to simulate, and the real data may just not be available. Mm. And so here we have to think about the inductive biases more.
158
00:28:53.172 --> 00:29:07.212
We have to add in the physics constraints, it cannot be just reliant on data, and that's where I think a little bit more thinking of the architectural design comes up. Uh, the other aspect is computational complexity.
159
00:29:07.572 --> 00:29:16.072
So think about language. Uh, it's just one dimension, and even there the context length, you know, we are getting to millions and we are struggling, right?
160
00:29:16.172 --> 00:29:40.592
I mean, on the other hand, now we are thinking about, uh, not just two D, three D, even four D, you know, three D and time, and if each of the dimension is even a few hundred grid points, which is where, you know, industrial scale starts at like, um, like a thousand grid points in each dimension, we are talking like hundreds of billions to even a trillion context length, right?
161
00:29:40.652 --> 00:29:47.212
So forget ever having a transformer for anything of this scale. All of the world's compute will not be enough.
162
00:29:47.252 --> 00:29:55.652
And first of all, they all have to be co-located to [laughs] be w- able to ever do this, so that's why we need other architectures. But I would push back a little bit, right?
163
00:29:55.792 --> 00:30:05.572
We have the vision and video language models, right? And they use-- They basically learn a mapping- But the resolution is very low. Mm. That's the key.
164
00:30:05.632 --> 00:30:12.912
Like, for the physical world, the resolution, what do I req- we require, I mentioned like thousand by thou- Yeah... and by thousand by thousand. Mm.
165
00:30:12.952 --> 00:30:25.432
So there, you know, the, if you count that, that's like already in hundreds of mill- billions. Yeah. So, so you know, we are not doing that high resolution when we think about images and videos currently.
166
00:30:25.732 --> 00:30:33.732
I have a friend here- And, and the video is also like autoregressive, so it's essentially only like you only need to do the next step. Right. Yeah.
167
00:30:34.132 --> 00:30:37.792
But you're learning, I mean, like generally you're learning a code book, right?
168
00:30:37.892 --> 00:30:56.972
So you have, you're kind of learning the bias of the latent space or the, the real world to the latent space and, and, and so that if there is a, like a compression that you can do from the physical world into the latent space, then, then, um, you know, these autoregressive techniques have been successful.
169
00:30:57.112 --> 00:31:08.572
Yeah, but the idea is, you know, a lot of these autoregressive and, you know, techniques for, uh, vision and video models are for mostly like, you know, looking good, right?
170
00:31:08.652 --> 00:31:17.992
So they are like not for very precise simulations and they're, uh, you know, be- having that higher resolution I- and details is really important.
171
00:31:18.512 --> 00:31:25.792
And so we need to at least take in the data of that high resolution, right? So we need to be able to process that and reason over them.
172
00:31:26.232 --> 00:31:48.232
And so this is where a lot of the bottleneck is because we, you know, cannot afford to just throw away everything and say, "Oh, let's just like have hundred grid points in each dimension or fifty grid points," because there just isn't enough detail to correctly model, uh, phenomena like fluid dynamics, plasma, how materials deform.
173
00:31:48.312 --> 00:31:56.416
So all of this requires high fidelity, and for that we need high resolution. My understanding, you have a thesis that AI needs,
174
00:31:57.396 --> 00:32:07.116
you know, to incorporate the physical world into it in order to scale and be accurate going forward. Many people have this thesis.
175
00:32:07.776 --> 00:32:21.096
You are somewhat unique in that you have several examples of applying neural operators to the physical world in this way, and then it seems like you're constructing a thesis around your experience here.
176
00:32:21.156 --> 00:32:33.176
So can you share with us some of the really interesting, um, and exciting looking things that you've done using neural operators and other techniques like this? Mm-hmm. Yeah.
177
00:32:33.236 --> 00:32:43.876
I mean, you know, for us, when we started with like neural operators for partial differential equations, but also more broadly, you don't even need to assume they're partial differential equations, right?
178
00:32:43.916 --> 00:32:47.176
It could be any spatiotemporal or data at multiple scales.
179
00:32:47.696 --> 00:33:05.056
So we, you know, set out looking for interesting examples, and one of them was like weather modeling because the weather data is open source, uh, it's, uh, available, uh, called ERA5 from the ECMWF, the European, uh, Agency for Global Weather Modeling.
180
00:33:05.496 --> 00:33:13.456
And so given that the data was there, we were like, "Okay, let's just go try it," right? Yeah. And that's the beauty of it. Whenever data is available, uh, it's really good news.
181
00:33:13.556 --> 00:33:20.096
But a lot of weather scientists did caution us back then. This was back in '21, and they said, "No, no, no, this is so difficult."
182
00:33:20.416 --> 00:33:29.296
You know, there have been like decades of like development in traditional weather forecasting, and that's very careful bottom-up physics-based modeling, right?
183
00:33:29.356 --> 00:33:36.116
So assuming, oh, this is the fluid dynamics, can you go predict the weather, uh, the next day and so on.
184
00:33:36.436 --> 00:33:46.456
And so that's how a lot of the thinking was that AI is just not gonna be able to beat, uh, the, you know, decades of work in weather modeling.
185
00:33:46.936 --> 00:33:55.396
Uh, but to our surprise, we just went ahead, we trained them, we used neural operators to be able to effectively capture the phenomena.
186
00:33:55.796 --> 00:34:11.456
And then now we, uh, to our surprise, we found that it's not only, you know, accurate, it's almost as close to the, uh, what the traditional weather models can do accurately, but also tens of thousands of times faster.
187
00:34:12.036 --> 00:34:24.776
So what would take a big supercomputer to run can now be run, and we only needed a consumer-grade like GPU. Like, you know, it was a small model, it fit very well, it's very fast, and it's accurate.
188
00:34:25.096 --> 00:34:33.476
And I think that just changed everybody's thinking. And after that, DeepMind, Huawei, many others followed us a year later, released their own models.
189
00:34:33.576 --> 00:34:45.876
We were the first to actually open source our weather model, FourCastNet, and do it permissively. So that's what allowed companies, um, weather agencies, everybody to build on us.
190
00:34:46.036 --> 00:35:09.396
And so, you know, it's been a really interesting revolution to see that the weather models are now out there, uh, weather agencies are adopting them, and it allows us to now have small weather agencies in the, uh, Global South, for instance, have the same kind of fidelity that very big agencies were in the past only able to do, right?
191
00:35:09.436 --> 00:35:11.436
So it's democratizing weather modeling.
192
00:35:11.876 --> 00:35:24.496
And so that's just one example of where there's been very quick, rapid progress and a paradigm shift in terms of saying that, "Oh, now we can have AI as a reliable way to do weather modeling."
193
00:35:24.976 --> 00:35:38.556
I saw that, you know, the, the models that you mentioned, you made an insight that nobody else had, and that people were able to devise other mechanisms to kind of follow behind you. But there was some sort of,
194
00:35:39.456 --> 00:35:42.036
uh, shift in thinking that was required here.
195
00:35:42.416 --> 00:35:58.156
And, um, was it, it, was it simply, um, we believe that there's enough structure in this data to learn and that people are just doing it wrong, and people found other ways to learn the structure, but that your method was very, um- So let me, let me clarify, right?
196
00:35:58.216 --> 00:35:58.736
Yeah. So there is...
197
00:35:58.996 --> 00:36:18.376
You know, first of all, the very first work was to just say that, you know, look, traditionally this has been done with, uh, so trying to solve partial differential equations, each time doing it again and again, whereas AI learns from data, learns patterns, and can be just as accurate but fast.
198
00:36:18.516 --> 00:36:23.416
Yeah. And then the next iterations was to say, you know, how do we make it even more accurate?
199
00:36:23.836 --> 00:36:35.976
And there's the aspect that, you know, there is the short-term weather, like what is predictable for the next two weeks, and then there's the long term, you know, going to subseasonal to ultimately climate modeling.
200
00:36:36.576 --> 00:36:47.996
And, uh, traditionally what people did was to have different models for these different scenarios. So there's a different kind of system that works for short term, another system works for long term.
201
00:36:48.476 --> 00:36:57.856
But to me, there's only one Earth. You know, if you want a foundation model, if the claim is that it should be able to do both very short term as well as very long term together.
202
00:36:58.296 --> 00:37:09.176
And that's where in FourCastNet3, the latest iteration of the model, we're able to do both. And that's because you also, you know, incorporate the spherical geometry of the Earth.
203
00:37:09.616 --> 00:37:26.596
And so with a lot of the other architectures that have been, uh, getting fairly good accuracies for the short-term weather, when you l- run them for longer term, when you run them for like several months to even year, even before that, it just very quickly blows up, right?
204
00:37:26.656 --> 00:37:29.796
Because it assumes the world is a rectangle, which it isn't.
205
00:37:30.296 --> 00:37:44.396
And so incorporating all of the geometry and that, uh, information into neural operators means that we can faithfully run the same model also longer term and, you know, make this into a climate model.
206
00:37:44.456 --> 00:37:55.488
This is where the Allen AI Institute has now built climate models based on our neural operator architecture, and that's the only one that works- As an AI emulator, right?
207
00:37:55.548 --> 00:38:09.828
None of the other architectures work for climate because climate requires us to assume the world is a globe and that if you are repeatedly rolling out you kind of ke-keep that information.
208
00:38:10.308 --> 00:38:20.828
Uh, whereas if it's like a narrow surrogate, that's what I consider a weather model, you just narrowly look at a few metrics, many different architecture should do the job, right?
209
00:38:20.908 --> 00:38:38.608
But if you are asking one architecture do a range of different tasks like a foundation model, that's where incorporating the geometry of the Earth, which is that it's a sphere, and using neural operators, an efficient way to do that enables us to accomplish that.
210
00:38:39.268 --> 00:38:50.568
Yeah. So ForecastNet, um, you said trained on fifty thousand data points. Can we just talk about like what does this look like? Like, what does the data input look like? What are you actually trying to predict from here?
211
00:38:50.608 --> 00:38:58.388
And then what is sort of the, like the large scale, um, you know, you said you, you're going from weather to climate. What does it look like to do that generalization?
212
00:38:58.527 --> 00:39:08.668
Because I could imagine if you have fifty thousand data points, these are, you know, some high resolution in North America, then, you know, they might even depend on the local geography.
213
00:39:08.748 --> 00:39:20.228
Like, like if you are always modeling, you know, Kansas, is this gonna transfer to, um, you know, let's say the Swiss Alps or something, and then does that transfer to, uh, you know, the Himalayas?
214
00:39:20.888 --> 00:39:25.248
So first of all, to clarify, we're training it on the global weather model, right? Okay. Mm-hmm.
215
00:39:25.288 --> 00:39:35.528
So we have all the information around the Earth, and then we are, you know, s-asking it to predict, like, given the current weather, uh, like say wind conditions- Yeah...
216
00:39:35.628 --> 00:39:47.408
humidity and so on, what happens, uh, in a, in an autoregressive way, and it's every six hours. So what happens in the next six hours and so on, and you roll out, and you train the model to predict.
217
00:39:47.948 --> 00:39:55.888
And, and so, you know, you can have potentially the same model predict forever, right? [laughs] So- Yeah. And but the predictability window is like the weather. Mm-hmm. Yeah.
218
00:39:55.908 --> 00:40:09.168
And if you want to go beyond, you have to do what we call ensembles- Mm-hmm... meaning you have like a probabilistic estimate of what happens, uh, uh, in several months to years, and that's how you get a climate model.
219
00:40:09.468 --> 00:40:20.608
Ah, okay. So you have, you actually have an ensemble of these local predictors, and then you use the, some sort of statistics on the ensemble to get a climate prediction? Yeah. I see.
220
00:40:20.648 --> 00:40:30.168
So you kind of like have several rollouts. Essentially, you have several trajectories of rollouts, and then you get- And so the efficiency- You get probabilistic... is actually pretty key here. Yes.
221
00:40:30.228 --> 00:40:37.088
And that's why that's the biggest bottleneck with traditional climate modeling, that it's so expensive to do even one single run.
222
00:40:37.128 --> 00:40:50.188
You have to do long trajectories, very, very high resolution and, uh, you know, that's why we don't have a lot of very high resolution ability to do climate change predictions, for instance.
223
00:40:50.368 --> 00:40:59.128
So how do you validate climate? So you do lots of rollouts, and you look at sort of how things evolve kind of in aggregate. I mean, Nissi said butter-butterfly effect.
224
00:40:59.268 --> 00:41:12.148
Locally, I think we can believe even a perfect climate model or perfect weather model would only give you maybe two weeks before it sort of, it's becomes non-computable. So, um, you do lots of rollouts.
225
00:41:12.328 --> 00:41:25.028
Um, there's some chaos. You, uh, you average all these things. How do you actually validate that this works over long enough times? Yeah, timescales. Yeah. Yeah. And, and, you know, it's a tricky question, right?
226
00:41:25.108 --> 00:41:36.408
So for instance, you have to kind of ensure that you satisfy all of the physical constraints, and if you just do a standard rollout, that is, you know, likely not gonna happen. Mm-hmm.
227
00:41:36.428 --> 00:41:44.128
And so like some of the ongoing research we're doing is how do you kind of enforce the right physical constraints as you do the rollouts? Mm-hmm.
228
00:41:44.168 --> 00:41:53.888
Like, you don't want to, you know, completely wash out the fine details because then it's not accurate. Uh, but on the other hand, if you keep them, you may be physically they're invalid.
229
00:41:54.348 --> 00:42:09.348
So this is still an open problem, and that's what makes it difficult that you want AI to be fast, and you want to be able to do these, uh, long climate simulations and at the same time be able to have full confidence in them.
230
00:42:09.388 --> 00:42:20.968
But these are things we are working on now. I don't know if this falls into weather or climate, but we recently just had one of the most extreme heat waves in the history of modern climate data, right?
231
00:42:21.228 --> 00:42:28.188
Um, in the, just right here in the, uh, kind of southwest United States. I'm wondering, were you-- did you...
232
00:42:28.828 --> 00:42:39.048
I, I don't know if you were involved in this regularly, but do you know if you or anyone actually modeled that or predicted that correctly? Yeah. Yeah. So we have-- I know I don't have information on this specific one.
233
00:42:39.068 --> 00:42:50.688
Yeah. But we've, uh, tested in our latest, uh, ForecastNet three model extreme weather events of all kinds, right? Mm-hmm. And that's the key, like, you know, that's where you need probabilistic answers.
234
00:42:50.748 --> 00:42:57.628
So having just one deterministic output and saying that this is the weather is not enough when we are looking at extreme events. Mm-hmm.
235
00:42:57.648 --> 00:43:07.468
So we need careful probabilistic calibration, and we show that we are able to capture those well. And I think that was the surprise, even in our very first attempt- Mm-hmm...
236
00:43:07.508 --> 00:43:20.708
that we visualized certain hurricanes and storms, and it was able to do well, which is very surprising because you would think that rare events are not something AI would do well, right? Right.
237
00:43:20.708 --> 00:43:22.628
It would do well on typical events.
238
00:43:23.028 --> 00:43:35.968
But I think this is where more broadly the lesson is the physical world may be more forgiving because, you know, where there are extreme events like hurricanes that have very specific physical signature, right?
239
00:43:36.028 --> 00:43:43.628
So it's like extreme, but in a very specific way. So maybe you don't need as many samples because the physical world has a lot of structure.
240
00:43:43.988 --> 00:43:49.888
And that's something we see this again and again, that there is a lot of structure in so many other examples.
241
00:43:49.948 --> 00:44:08.168
You know, talk about plasma and fusion reactor- Uh, you know, we barely have a few thousand samples, but we are able to accurately predict, uh, events like disruption, uh, very well, and we are able to do that a million times faster than what traditional simulations were able to do.
242
00:44:08.668 --> 00:44:20.648
To me, yes, all these sound very surprising, but it's because I think the nature helps us a lot. You know, it has a lot of latent space structure [laughing] Yeah. That, uh- Structure, yeah.
243
00:44:20.788 --> 00:44:31.628
That I don't think traditional numerical methods are able to uncover because they are focusing more on correctness, that in any scenario you should be able to solve these equations.
244
00:44:32.068 --> 00:44:44.428
On the other hand, with AI, it's learning from data, it's uncovering the structure, it's uncovering how easy or s- kind of tractable these problems are, and that's what we are seeing in many cases. Yeah.
245
00:44:44.508 --> 00:44:55.908
I, I've had of a, a similar analogy. So from, you know, my own da- domain is probably closer to computational biology, but, you know, AlphaFold is the obvious, uh, like, really exciting development in the community.
246
00:44:56.508 --> 00:45:05.728
So the solving protein structure prediction and of course, all the caveats of what was actually solved, well, I think we discussed this in a, in a previous episode with, uh, the Boltz team.
247
00:45:06.188 --> 00:45:08.648
Encourage listeners to, to listen to that if they want more.
248
00:45:08.808 --> 00:45:26.048
But one of the, I think, key points about protein structure is that it really is constrained by physics, and that's why it was, in some sense, one of the, the few big wins in the field of biology, which is otherwise very complex, and we've had trouble, generally speaking, making a lot of success.
249
00:45:26.728 --> 00:45:39.608
And it seems like problems which are solved by differential equations or, I'd say, modeled well by differential equations, have a lot more, um, room for also integrating these techniques, um, as a general form. Yeah.
250
00:45:39.628 --> 00:45:43.088
Absolutely. I don't think there was a question there. Yeah. Absolutely. Yeah, editorializing. Yeah.
251
00:45:43.128 --> 00:45:58.108
So, so, so getting back to my question, um, we have climate or weather and climate, we have plasma, we have, um, uh, biology. Y- and I know that y- you, you prepared for us a few visualizations. Oh, yeah.
252
00:45:58.328 --> 00:46:07.208
Can you, can you just share with us, um, what does that look like? You know, what are the, so the visualizations and then what is the thread that runs through here?
253
00:46:07.248 --> 00:46:16.968
And I think maybe the re- the listeners will already have a hint about that, but I'd be really excited to see that. Yeah. Yeah, I can certainly share some of them.
254
00:46:17.108 --> 00:46:25.788
Uh, I mean, this one is just kind of showing that, you know, we have world at different scales, right?
255
00:46:25.848 --> 00:46:39.128
And these are examples of phenomena happening at different scales, from atomic to protein to even planetary scales, like the weather we talked about. And, you know, we need to capture all of that.
256
00:46:39.188 --> 00:46:47.448
That's what neural operators are, are designed to do, uh, and to kind of, you know, feed in data these different scales.
257
00:46:47.948 --> 00:46:55.708
And that's really, like, the aspect that makes a lot of the physical world problems hard, that need for fine scale.
258
00:46:55.768 --> 00:47:07.988
You know, we talked about how a lot of traditional computer vision video models are just designed to make things look visually good, and that requires low enough resolution.
259
00:47:08.048 --> 00:47:21.028
It's tractable, autoregressive, it's enough that, you know, it works out, it's short enough videos. But that's not how a lot of the physical simulation for, you know, industrial scale, high fidelity work.
260
00:47:21.168 --> 00:47:26.868
You really need high resolution. The, you know, ca- the atmosphere is one example.
261
00:47:26.948 --> 00:47:36.368
Like, you, depending on the resolution you observe, different phenomena can be captured, so you just miss that out if you don't have that high resolution.
262
00:47:36.708 --> 00:47:42.328
And now the question is, with AI, can we do this much faster than what we could with traditional simulation?
263
00:47:42.928 --> 00:47:56.808
And this one with neural operators is kind of showing that, you know, if you use the standard neural network and you had a fixed number of pixels like you're seeing on this side, and you zoom in, it gets blurry, right?
264
00:47:56.868 --> 00:48:02.648
That's the end of it. There's nothing beyond those, uh, fixed resolution that you can capture.
265
00:48:03.088 --> 00:48:22.848
But the idea is with neural operators, because it's a function space representation, meaning you can keep zooming in, you can add it the relevant details either by giving it data at higher resolution or physical constraints at higher resolution, you can kind of bring that, uh, multi-scale phenomena together.
266
00:48:23.088 --> 00:48:32.608
So where, where are the physical constraints? I mean, I assume physical constraints here are, like, local simulating, um, you know, fluid equations or some sort of- Yeah... hydrodynamic. Could be, right?
267
00:48:32.708 --> 00:48:45.268
So it could be of any nature. Mm-hmm. The idea is now you can add, like, conservation laws- Mm-hmm... for instance, in an incompressible fluid. Mm-hmm. You can add, like, material deformation, like how things stretch.
268
00:48:45.688 --> 00:48:57.268
So, uh, or it can be a full partial differential equation. Mm-hmm. So that's also an interesting question we've been researching, how is the curriculum of different physics, like you mentioned. Mm-hmm.
269
00:48:57.668 --> 00:49:09.048
You know, some physics may be very hard to impose or add as a loss function. Mm-hmm. Others may be easier. So you also need to kind of, you know, think about what, what to impose, you know?
270
00:49:09.248 --> 00:49:18.508
So, and the intuition is here is that when I'm adding a physical constraint, I'm adding it to the loss function. Is that more or less what's happening? Yeah. Because that's what is tractable, you know? Yeah.
271
00:49:18.528 --> 00:49:21.208
Making it a hard constraint is- Right... not tractable. Right.
272
00:49:21.568 --> 00:49:32.500
Whereas adding it as a loss function and, of course, there's still the balancing of that loss with the data we have, so we have to, you know, uh, do that in an appropriate way. Right. Yeah.
273
00:49:32.640 --> 00:49:46.040
So as I mentioned, the, this is the example of the weather model where, uh, here we are showing how, uh, we are able to capture, like, atmospheric rivers, which is the phenomena we see here in California- Every winter, yeah...
274
00:49:46.080 --> 00:49:54.780
you know, resulting in large storms. I think we're, we have one expected later this week, so [laughs] We'll, we'll hold you to that. Yeah.
275
00:49:54.840 --> 00:50:02.200
[laughing] Uh, so the idea of, like, you know, why show this is this kind of global phenomena, right?
276
00:50:02.280 --> 00:50:16.800
These are, like, thousands of miles wide, so you really need non-local models that capture, uh, these very, uh, large span phenomena and do that accurately, and that's what our neural operators are able to do.
277
00:50:16.980 --> 00:50:26.480
So, so the training data for this is... You were talking this a bit before, but I'm still, like, wondering what is this l- this is weather satellites, or is there ground-based, uh, data?
278
00:50:26.540 --> 00:50:35.740
Is it some hybrid of the, the two? It's, it's kind of a combination of different- Mm-hmm... sources, so it's what we call reanalysis data. So this is historical weather data- Mm-hmm...
279
00:50:36.140 --> 00:50:47.320
that is, in a way, reanalyzed, meaning that the sat-raw satellite data is combined with essentially what the physics solvers- Mm-hmm... tell you together a-assimilated.
280
00:50:47.880 --> 00:50:52.220
And so this is, uh, made available by the weather agencies, and we can train on them.
281
00:50:52.740 --> 00:51:03.120
So you're saying that they take a low-resolution data set, which is compiling all of the world's data set, you know, all of our meteorological data we have across the world- Mm-hmm...
282
00:51:03.200 --> 00:51:12.520
uh, and then they do short time simulations using physics-based, you know, classical techniques to, to fill in the details. Yes.
283
00:51:12.620 --> 00:51:22.220
Uh, and you can do that over short time spans, but as you go farther, it breaks down very quickly. So you're amortizing that across all the...
284
00:51:22.640 --> 00:51:30.200
Everyone would have to do that, and so somebody does it, and then you are able to take advantage. Yeah. I mean, this is data, right, that's already prepared.
285
00:51:30.260 --> 00:51:37.700
But the idea is already this data assimilation with physics kinda makes our model physics informed implicitly.
286
00:51:37.780 --> 00:51:47.420
So it's able to kinda, you know, k-keep that information, and that's why maybe that's one reason maybe it does well on even extreme weather events.
287
00:51:48.060 --> 00:51:58.440
Yeah, so this is just showing that we, our model is available in ECMWF, which is the weather agency, like, you know, the European weather agency.
288
00:51:58.560 --> 00:52:06.380
Uh, and so this was launched, like, more than two years ago, but, you know, I think fall twenty twenty-three. So, you know, s-
289
00:52:07.380 --> 00:52:21.220
I think ECMWF making these AI-based weather models available to the public, to me, was a very big step because that's where, you know, everybody could see what's happening. There were several hurricanes.
290
00:52:21.280 --> 00:52:28.060
Like, for instance, there was Hurricane Lee, and that's where the public could see what are these weather models doing.
291
00:52:28.160 --> 00:52:40.780
For instance, our ForecastNet was able to correctly predict that the hurricane m-making the landfall several days earlier compared to the standard, uh, weather forecasting models.
292
00:52:41.200 --> 00:52:53.680
And so the idea that these models could be very good for extreme weather events, uh, and do early prediction, you know, both for human lives, for economic costs, is a very big deal.
293
00:52:53.740 --> 00:53:03.120
And so that's when the public kind of got a lot more, I think, buy-in, uh, and from weather scientists because of how well it was doing in these events.
294
00:53:03.700 --> 00:53:14.500
And this is what I was talking about in a ensemble prediction, both for extreme weather or if you're thinking about climate. It's not just about looking at one trajectory, right?
295
00:53:14.580 --> 00:53:23.640
Because, you know, for unless you're somebody with a Sharpie somehow saying where the [laughing] hurricane is gonna go. [laughing] No, pun intended.
296
00:53:24.220 --> 00:53:41.240
But, you know, what you really want is the probabilistic prediction, meaning, you know, I'm gonna try different adding noise levels to my initial condition, what the, when the weather, uh, when the hurricane is forming in the Caribbean, I'm gonna add some noise because anyway, it's noisy.
297
00:53:41.280 --> 00:53:45.040
I don't know truly what the measurement there is.
298
00:53:45.420 --> 00:53:56.940
And then I'm gonna look at what happens to the possible hurricane tracks, and then I can come up with a probability of landfall in different regions, and that's how I can do risk assessment.
299
00:53:57.440 --> 00:54:04.000
And so this is where it gets even more expensive for traditional weather models because you have to run all of these ensembles.
300
00:54:04.340 --> 00:54:17.500
And now AI weather models being so fast, tens of thousands of times faster, means we can now do very large ensembles, and this is a very big improvement in terms of what we can do for risk assessment.
301
00:54:18.000 --> 00:54:29.720
Have you gone through and done, let's say, looked over the historical, uh, hurricane maps and then tried to do ensemble predictions and calibrated how often your predictions are, like a code? Yeah. Yeah.
302
00:54:29.780 --> 00:54:41.680
So in ForecastNet- Mm-hmm... three paper, there are, you know, uh, we have metrics of, like, extreme weather events- Mm-hmm... and ensemble prediction. And in fact, we've trained the model to do good ensemble prediction.
303
00:54:42.080 --> 00:54:53.120
Mm-hmm. And so this is where the calibration matters, uh, for, for these kind of events. Yeah. What was the sort of key insights or developments in ForecastNet three in versus two versus the first version? Yeah.
304
00:54:53.180 --> 00:55:02.060
So the first version was kind of the, you know, the using, like, the Fourier neural operators, but we didn't incorporate the spherical geometry, right?
305
00:55:02.140 --> 00:55:09.380
In this next version, we said, "I think, you know, it's important that the world is a sphere," because first of all, otherwise it's distorted- Mm-hmm...
306
00:55:09.420 --> 00:55:17.180
so you kind of are not predicting, uh, it in a- Sorry, weird dumb question, but if it wasn't spherical, what did you do? Like a Mercator projection or something?
307
00:55:17.440 --> 00:55:23.560
Yeah, the standard, like, kind of the, you know, like as you-- I, all, and all the other weather models do the same. Oh, really? Okay. Right?
308
00:55:23.620 --> 00:55:31.700
So they just kinda have the standard projection and then, uh, you know, predict the weather, and which is okay for short-term prediction.
309
00:55:31.760 --> 00:55:42.780
But when we, you know, our goal was to have the same model also do longer term, and that's when incorporating the spherical geometry added this additional stability. We could do longer roll-outs.
310
00:55:43.160 --> 00:55:48.540
And then in ForecastNet 3, the idea was it's not just about deterministic prediction.
311
00:55:48.860 --> 00:55:59.420
We want to get ensemble predictions right, so we have to train them based on this objective that we get the probabilistic predictions correct as well. How long are you predicting out?
312
00:55:59.940 --> 00:56:11.480
Um, and how many roll-outs are you doing? Yeah, so the... I know rollout is how long you predict, right? So each step is six hours, and then you predict for however long you want.
313
00:56:11.480 --> 00:56:23.420
[laughs] You know, you just have to roll out. Sorry, how many, how many, uh, examples in the ensemble do you have? So, a-and again, that's our choice. We can have- Yeah... like ensembles of different levels.
314
00:56:23.480 --> 00:56:34.420
So we-- I think it's like a few tens or something like is what we are currently, you know, shown, but you can do much larger too. And that's adequate to get out how far?
315
00:56:35.100 --> 00:56:44.180
My intuition is, the longer you want to predict, the more, uh- So not necessarily. Okay. It's really like about, again, calibrating the ensembles and ensuring- Mm-hmm...
316
00:56:44.220 --> 00:56:58.740
that they have the right spread rather than- I see... you know, so, so- Okay. So may-m-- So you have tens of these, um, models or examples in your ensemble, and that even with a very, very long roll-out, that's adequate.
317
00:56:59.120 --> 00:57:05.260
So again, like, you know, there's, as I said, a lot of still outstanding questions to do very, very long roll-outs. Mm-hmm. Right?
318
00:57:05.300 --> 00:57:14.340
Because you, you do need to incorporate like the physical constraints in a way to ensure that. That's something that we are actively researching now.
319
00:57:14.840 --> 00:57:26.480
Uh, but, uh, these models that we have are able to do the longest roll-outs compared to any of the other weather models that completely ignore spherical assumption and a range of other things.
320
00:57:26.980 --> 00:57:34.040
So when you say incor-incorporate, um, physical laws for climate over long times, I mean, what does that look like?
321
00:57:34.080 --> 00:57:43.180
Because there's a lot of, like, local conservation, which may be just broken if you take an ensemble of average, even though any given snapshot is, respects that.
322
00:57:43.220 --> 00:57:49.020
Or- No, the idea is to make sure you, uh, look at each ensemble- Mm-hmm... member and it respecting the physics. Okay, okay.
323
00:57:49.340 --> 00:57:58.400
You're not assuming the ensemble, uh, you're not deriving a, like, coarse-grained, like, equivalent of a probability- No... or something. Okay. Because then you would lose that- Yeah...
324
00:57:58.500 --> 00:58:07.520
you know, uh, resolution and the fidelity. Okay. That's a little confusing to me. Wait, so you're... So it's not an average. Well, how are you combining the ensemble?
325
00:58:07.640 --> 00:58:17.320
No, you are doing the average, but you're predicting each one. Oh, you're predicting each one separately. Yeah. Okay. Yeah, yeah. So each one is independently satisfies these constraints. Yes. But the ensemble- Mm-hmm...
326
00:58:17.600 --> 00:58:28.240
does not. Yeah, which is- And, and so that's how you ensure physical validity. I see. Okay. Yeah. So when you go on a sphere, you operate in, do you use basically spherical harmonics or some sort of- Yes. Okay, yeah.
327
00:58:28.680 --> 00:58:37.580
You have a, uh- Yes... spherical basis for it. Exactly. Which is actually very natural with Fourier netflow. Exactly. It's probably much harsher [laughs] if you're doing- Yeah... um, other. Yeah. Exactly.
328
00:58:37.620 --> 00:58:48.460
So that's where the, you know, like the Fourier saves us. Yeah. [laughing] Can incorporate these geometries well- Mm-hmm... and, and so I think be very faithful to, you know, what the globe is.
329
00:58:48.460 --> 00:58:58.200
[laughs] Yeah, arguably, it's much more natural- Yeah... than a, than like a Mercator projection- Yeah... or whatever other. Yeah, which [laughs] you know- Yeah... is like Greenland becomes huge. [laughing] Right.
330
00:58:58.220 --> 00:59:00.500
Exactly. Yeah. So that is a different story.
331
00:59:00.600 --> 00:59:14.440
But, uh, but yeah, but I think th-this is where I think the aspect of, you know, more broadly incorporating more of geometry and information about the domain becomes a lot more important in, for the physical world, right?
332
00:59:14.500 --> 00:59:25.420
So this is me again emphasizing that we need to incorporate more of the structures because one is the data is limited, and the other is a lot of what we are asking is extrapolation, you know- Mm-hmm...
333
00:59:25.540 --> 00:59:36.020
to go beyond than what it's trained on. You know, we're just training it to predict the next six hours and maybe do a little bit of multi-step fine-tuning for autoregressive roll-outs, right?
334
00:59:36.100 --> 00:59:45.420
So we are not training it to do very long, like a climate simulation, because that's just too expensive. But we hope magically it works well.
335
00:59:45.560 --> 00:59:59.840
[laughing] And it cannot if you just say, "I'm just gonna put a standard transformer or whatever else there," and it won't work out. So we added more of the domain constraints like spherical geometry.
336
00:59:59.920 --> 01:00:10.240
We add in maybe more of the physics in certain ways, and that's where it becomes more interesting algorithmically as well. You know, there's, uh, more in model design here.
337
01:00:10.720 --> 01:00:21.980
So the timescale that you train on, how, how long is that? To predict for the next six hours. Oh, j-- so only six hours? Yeah. That's abou- And a little bit of multi-step fine-tuning like together, like- I see.
338
01:00:22.020 --> 01:00:28.880
Yeah, yeah, I said that. Okay. Understood. So which is very surprising. [laughs] Yeah, that is very surprising. I would have expected it was weeks or months or whatever. Yeah.
339
01:00:28.920 --> 01:00:40.960
No, and then it kind of just works well even for like now we are showing for several months that it's able to do that. The number of steps is on- Hundreds or thousands.
340
01:00:41.000 --> 01:00:53.760
Yeah, and, and that your, your Fourier basis, you-- it's, it's many times the, the, the sort of base harmonic. I mean, this is like, that's in space, right? Yeah. So we're talking roll-out is autoregressive with time.
341
01:00:54.100 --> 01:01:06.560
No, but like in, in time, I'm, maybe I'm misunderstanding here, because it's a, in Fourier domain, right? No, no, in time it's not. That's what I'm saying. It's autoregressive in time. Oh, oh, I understand. Okay.
342
01:01:06.940 --> 01:01:12.740
So it's space, it is. Yeah, yeah. Yeah. Interesting. Yeah. Okay. What's the, uh, angular resolution- At least in, in this scenario.
343
01:01:12.780 --> 01:01:23.920
In other cases, we also have in-- time is also represented in the Fourier domain, and that's a question as well, can we do that? Right. But in this example, it's autoregressive. Got it. Yeah.
344
01:01:24.000 --> 01:01:30.940
Um, what's the angular resolution that you use for, for this, um, in, in, you know, in the spherical version? Yeah.
345
01:01:31.000 --> 01:01:44.480
So it's, so all of the data that's available is like, uh, I think a quarter, like point two five degrees. So that's- In terms of L maybe, or s-- I mean, in terms of like a spherical harmonic frequency- Uh...
346
01:01:44.500 --> 01:01:49.740
or, and I'm just curious. So this is like, uh-- Oh, you mean like how many modes we utilize? Yeah, I guess.
347
01:01:49.800 --> 01:01:58.140
I think we, so for that resolution, and we essentially utilize, I think most of it, only a few of them will leave out. I forget the details, but...
348
01:01:58.560 --> 01:02:06.880
I'm just curious, like what is the actual angular resolution on the globe that you are resolving to? Or maybe just the a- the, the physical resolution. Yeah. I mean, and that's what, uh- Yeah...
349
01:02:06.920 --> 01:02:17.400
that, because that's the data we get, right? So right now the data we get is like a quarter, like, uh, so point two five degrees. Totally. Oh, okay, like, uh, point two five solid angle.
350
01:02:18.068 --> 01:02:29.368
I think kind of comes out to like, uh, I don't know, seven hundred by few thousand, like, guide resolution. Mm-hmm. So but this is already standard, like, kind of processed. Yeah, yeah.
351
01:02:29.748 --> 01:02:36.268
And just, uh, trying to understand, like, how large of the basis do you need to represent this? Yeah, I mean, that's really depends on the resolution.
352
01:02:36.308 --> 01:02:48.128
And, and the idea is, you know, right now our weather data is just limited by this resolution. But if you could, you know, you could, like, kind of do synthetic climate simulations of even higher resolution, right?
353
01:02:48.188 --> 01:02:58.588
And that's kind of the next thing on how to combine these together. Do you think you can predict with super resolution be-- and basically resolution lower than the data provided?
354
01:02:58.908 --> 01:03:09.428
Again, like, yes, we can always predict them with- Sure... the neural operators. Uh, but you know, you do wanna incorporate more of the physical constraints to ensure that they are valid.
355
01:03:09.828 --> 01:03:15.688
Okay, so at, um, can we talk about- Yeah... some of the other, uh- Yes, yes, I know... technologies? There's a lot.
356
01:03:16.328 --> 01:03:27.768
Uh, so this is just showing, like, how, you know, the, what I described that the on the left wh- where the world is being assumed, it's a rectangle, it blows up very quickly. Mm-hmm.
357
01:03:27.828 --> 01:03:37.408
And on the right, because we assume the world was a sphere, it kept rolling it out, and it kept being stable. So- I still see a little bit of a singularity there- Yes... at the right [laughing].
358
01:03:37.448 --> 01:03:44.268
And it's still like, uh, you know, so the idea is, yes, because it's a very long rollout- Sure. Mm-hmm... and we have no guardrails of physics. Mm-hmm.
359
01:03:44.308 --> 01:03:56.628
We are not, you know, kind of projecting it to the right physics, right? This is full extrapolation. But the idea is the sphere assumption stabilizes it to a much greater- Yeah... extent.
360
01:03:56.648 --> 01:04:03.268
Compared to the left, it's, uh, much better. Yeah. Yeah. [laughs] But if you're, you're in the South Pole, you're still not gonna get exactly that. Exactly. So the poles are the hard part, so.
361
01:04:04.108 --> 01:04:19.768
So this is an example of the fusion reactor. So this is a tokamak, and we are able to model the complex, uh, plasma evolution and do this a million times faster than what we could do with traditional simulations.
362
01:04:19.988 --> 01:04:26.068
And, uh, this was the, in a way, we're creating a digital twin of the plasma, right?
363
01:04:26.147 --> 01:04:39.068
And then we can, you know, do further things like right now we are, as a next step, looking at, like, control, but with a full valid physics, like being able to prevent disruptions ideally and make fusion sustainable.
364
01:04:39.408 --> 01:04:45.428
So are you simulating MHD equations here or... Sorry. Yes. Magnetohydrodynamics equations? Yes, yes. Okay. Yeah.
365
01:04:45.448 --> 01:05:00.188
And then so for context, disruption in this case is this, uh, phenomenon which plagues, which plagues plasma physicists, where at some point, your entire plasma collects in a little tiny beam and then shoots a strong, you know, right through your containment vessel and blows it up.
366
01:05:00.308 --> 01:05:07.248
Yeah, and it can damage- Yeah... the reactor. Yeah. And that's the, that's a big bottleneck because then you have to kind of shut it down before that happens. Yeah.
367
01:05:07.268 --> 01:05:19.308
And then plus, it's no longer possible to have a sustainable fusion. Yeah. So there's a lot of open challenges here, but the idea is, you know, it's very expensive to go do physical experiments. Mm-hmm.
368
01:05:19.328 --> 01:05:31.368
The more you can capture that in a digital twin, but ensure physical validity, the more you can even do design and other considerations, uh, in the digital realm.
369
01:05:31.908 --> 01:05:36.028
You know, we can hopefully make advances, and these are the first steps towards that.
370
01:05:36.388 --> 01:05:50.468
The goal is that if you have one of these events, that you can somehow adjust the, the, um, the magnetic field so that it contains that and, and stabilizes. Yes. And that's the next step we are doing now.
371
01:05:50.528 --> 01:06:02.988
We are looking at, like, designing both the control and the simulation together. Nice. Are you working with a specific lab? I'm just curious. So this one was with the UK Atomic Energy Agency. Okay, okay.
372
01:06:03.028 --> 01:06:14.868
And now we're also working with a few others here in the US as well. So we are- Yeah... you know, kinda- Sure... getting the information from many different approaches of fusion itself. So this is the tokamak.
373
01:06:15.268 --> 01:06:22.228
We're also working with stellarators. We are working with different [laughs] approaches. The stellarators are- Yeah... are tricky. Yes. They're, yeah.
374
01:06:22.268 --> 01:06:32.908
But the idea is ideally, you know, like, our goal is to be able to design them in the digital twin. So can we come up with good designs that would make it maybe more practical?
375
01:06:33.428 --> 01:06:47.348
And so- And so that's, I think, also a good thing as an AI person and much more like, you know, agnostic and not picking a winner beforehand, right? Like, I like to work with different approaches, you know, uh, and see
376
01:06:48.388 --> 01:06:55.728
whether AI can accelerate all of them, and then we can kind of- Mm-hmm... you know, s- not prematurely rule out one approach over the other.
377
01:06:55.808 --> 01:07:07.288
So that's what AI enables us to be more kind of taking risks and exploring different approaches as opposed to in the physical world, trying to build any of these.
378
01:07:07.468 --> 01:07:14.208
You kind of have to cut a lot of the risky ones and say, "I'm only gonna do this because this is the most likely to work."
379
01:07:14.688 --> 01:07:21.688
I noticed over your career, you started out spending a lot of time on, you know, really theoretical foundations and mathematics of machine learning.
380
01:07:22.208 --> 01:07:32.868
And maybe, I don't know, something like six, eight years ago, you started working, really working a lot on applications and branching out in a diverse set of problems.
381
01:07:33.428 --> 01:07:44.488
What sort of prompted that shift in your, your, your approach and what you're looking at? I mean, so you're still working on very hard math problems as well, like, for example, the, the TorchLean work.
382
01:07:44.888 --> 01:07:54.128
But, um, the, the applications have really grown, and I'm wondering what, what prompted that? And, like, um, what were some of the lessons you've learned since then? Sure. Yeah. Sure.
383
01:07:54.188 --> 01:07:58.828
I mean, to me, it's like, you know, I feel like I've grown along with AI, right?
384
01:07:58.908 --> 01:08:18.508
So when AI was, you know, in this, where neural nets were not working because there wasn't enough data and all kinds of other reasons, you know, then you kind of have to build the theoretical foundations and try to hope that that leads you to a place where, you know, you get algorithms to work, right?
385
01:08:18.568 --> 01:08:26.728
And, and, and, and, you know, back then, like tensor methods was with that idea that, uh, you know, pre-deep learning, we still want a structure.
386
01:08:26.768 --> 01:08:38.136
We have probabilistic models like latent Dirichlet allocation for topic modeling and- Solving those were hard, but now tensor methods gave us a way to be very practical.
387
01:08:38.216 --> 01:08:43.096
It's parallel and can be done at large scale, but still has nice theoretical basis.
388
01:08:43.156 --> 01:08:57.936
So that was where, you know, was starting off, and then as deep learning started taking off and we could see that it works well in practice, and yes, there is a little bit of maybe theoretical understanding, but not a whole lot because of the way how complex it is.
389
01:08:57.996 --> 01:09:13.676
To me, theory should not be a constraint, right? It should be an enabler. And so that's where a lot of, like, the exploration was all to make this work well in practice and over into Amazon Web Services, then NVIDIA.
390
01:09:14.076 --> 01:09:23.196
So really like making things work at scale and really kind of getting hands dirty, right, was kind of like where a lot of the development is.
391
01:09:23.256 --> 01:09:35.256
And now I see a full circle because a lot of da-- purely data-driven approach is, in a way, seeing saturation, right? So now we want to ask, okay, either make them more hardware efficient, right?
392
01:09:35.336 --> 01:09:44.756
There's a lot of now room to kind of say, "Can we now, you know, make them much more energy efficient or hardware efficient?" So that's one aspect.
393
01:09:45.096 --> 01:09:57.416
But the other is areas like this where in the physical world, we don't have enough data. We are asking for hard extrapolation. You know, we want to think of doing discovery. By nature, it's about extrapolation.
394
01:09:57.476 --> 01:10:01.476
So we will never have data about a new discovery, right? That's by definition.
395
01:10:01.996 --> 01:10:15.236
And so there we need to again go back to thinking in principled ways, in the-- whether it's architecture design, algorithm design, the right loss functions. So we need to be much more mindful.
396
01:10:15.336 --> 01:10:27.236
So I see that coming a full circle because all of the things that work with deep learning, let's take them, but make them a bit more principled. There are several other applications which seem very natural.
397
01:10:27.636 --> 01:10:31.856
I'm wondering if you've worked on these or did I just miss some papers? If I did, I'm sorry.
398
01:10:32.396 --> 01:10:42.256
Um, so some examples are design of, like, electromagnetic circuits, um, I think is a big one, uh, or maybe not a big one, but I think will be coming up in the near future.
399
01:10:42.676 --> 01:10:54.576
Design of, let's say, materials, um, design of, let's say, dissipation and heat sinks, uh, or, um, or any sort of like fluid flow as I'm going through. What differential equations do I know?
400
01:10:54.616 --> 01:11:08.956
Electromagnetism, I know diffusion equations, um, you know, M-MHD. Yeah. Um, I'm wondering some of the other domains that are, um- Yeah, I mean, to me, there is just endless possibility, right? So there- Mm-hmm.
401
01:11:08.976 --> 01:11:15.436
You know, as like you can just have this work on any data and we have several other examples.
402
01:11:15.536 --> 01:11:25.556
So this was like, you know, being able to ask, can we sequester carbon dioxide underground and model, uh, how carbon dioxide, uh,
403
01:11:26.636 --> 01:11:35.996
expands or, you know, what is the pressure build-up in these reservoirs and, uh, you know, can we kind of model how they migrate over several decades?
404
01:11:36.476 --> 01:11:41.516
And so this one we were able to do much faster than what traditional simulations could do.
405
01:11:41.936 --> 01:11:50.096
I mean, the other aspect is being able to do all kinds of geometric shapes like, you know, being able to model aerodynamics in cars, planes and so on.
406
01:11:50.656 --> 01:12:03.956
And, uh, so again, this is a nice example of a latent space because you can transform a car or any other shape to a donut and then model on the donut and then transform the donut back to the car.
407
01:12:04.096 --> 01:12:09.416
So- You didn't turn it into a coffee cup. [laughing] Isn't that the classic joke? Your donut in your coffee cup. Exactly.
408
01:12:09.476 --> 01:12:23.996
So the idea of like a latent space to handle all kinds of different geometries and, and be able to capture the physics there in the latent space well means we can now have a model that generalizes across a lot of different geometries.
409
01:12:24.476 --> 01:12:39.356
Am I understanding that the, maybe the larger vision here is that, uh, you can train a foundation model in the sense of being able to model many different physical phenomena with the same model.
410
01:12:39.416 --> 01:12:56.436
And so you may fine-tune or there may be some kind of prompt that you give it to un-- have it understand the particular geometry, but that, that, that you, you know, sort of on all these different physical problems you train and then, and then you have your particular one and you're able to model that very effectively.
411
01:12:56.496 --> 01:13:03.216
Yeah. I mean, that's really the future, right? Because we have foundation models for language, maybe vision, but not for physics.
412
01:13:03.296 --> 01:13:12.176
So, you know, the idea is instead of like right now what we've seen are narrow surrogates, and we're trying to broaden their scope more and more.
413
01:13:12.336 --> 01:13:22.856
But ideally, we have much broader models that can work on a range of phenomena, but also multiphysics. So not just have like one single physics, but coupled physics.
414
01:13:23.116 --> 01:13:30.356
The real world has all of the physics kind of [chuckles] coming together in coupled ways. So can we bring all that together?
415
01:13:30.796 --> 01:13:40.336
So that's one aspect, like, you know, have foundation models that can do desi-- that can do simulation. But the other aspect that's really interesting is the inverse problem, right?
416
01:13:40.376 --> 01:13:44.976
So can I now not just simulate but ask what is the best design?
417
01:13:45.416 --> 01:14:06.176
And then these kinds of like, uh, models can like do simulation, but you can even do that implicitly and come up with the best design rather than in the earlier era of it was humans trying to come up with design, then you go and try to simulate or go to the wind tunnel, whatever physical testing and validate that.
418
01:14:06.216 --> 01:14:17.116
But now you have AI come up with optimized designs, but you have the guardrails of physics, so you have models that are accurate in physics, you have the confidence they work well.
419
01:14:17.576 --> 01:14:20.936
So you're kind of able to do that as well in the same model.
420
01:14:21.536 --> 01:14:37.336
Is there a reason-- Have you seen any evidence that you talked about these like sort of multiphysics being able to transfer or that, that, uh, you may be able to generalize to, um, sort of unseen physics? Mm-hmm.
421
01:14:37.996 --> 01:14:45.136
So, I mean, you know, like the physics by nature, if it's completely unseen, it's not possible to transfer, right? [laughs] Sure.
422
01:14:45.176 --> 01:14:51.176
I mean, I mean, if you're saying that we're going beyond the standard model- No, no, no, no, no... there's absolutely no data, that's not possible. Yeah.
423
01:14:51.216 --> 01:14:55.756
Uh, but if you're asking about like, you know, for instance, like I...
424
01:14:55.996 --> 01:15:10.156
You know, there is the like say I've like, you know, sh-shown it examples of like just how the heat propagates and the other examples of how the material like stretches and now there's coupling.
425
01:15:10.215 --> 01:15:21.336
Like, because of heat there's also stretching or kind of the joint phenomena. You could like now hope to fine-tune with much fewer samples because it kind of individually knows this phenomena.
426
01:15:21.856 --> 01:15:27.916
Then combining them together, maybe it can't do it from scratch because that's still too much to ask.
427
01:15:27.976 --> 01:15:38.916
It's highly nonlinear and coupled, but it can do it with fewer examples, and we've seen evidence of that in a lot of our papers that you're able to kind of essentially build up a curriculum.
428
01:15:38.996 --> 01:15:57.616
And that's what we see again and again in many of these examples that, you know, the real world we can kind of control a lot of curriculum and say, you know, let's kind of build in like modules and put them together, and that's what it now allows us to do in a systematic way here.
429
01:15:58.256 --> 01:16:03.256
I guess the design aspect, I don't know if we wanted to show, uh, very quickly.
430
01:16:03.336 --> 01:16:18.176
So this one was like, you know, looking at like designing the mask for inverse lithography, meaning now there's an inverse design problem, and we are also able to do that for designing gates and quantum dots.
431
01:16:18.296 --> 01:16:21.216
Uh, uh, this is like nonlinear photonics.
432
01:16:21.576 --> 01:16:40.696
And all of this, what is common is the idea that, you know, there's a forward model that is simulating the physics, but now what we want is the inverse design, like the problem that, uh, of we can optimize the best design, and humans are usually not good at this, right?
433
01:16:40.756 --> 01:16:51.656
[laughs] We are not good at like looking at highly nonlinear phenomena and say, "Oh, somehow maybe this combination of all these gates coming together helps pull the electrons together in a quantum gate."
434
01:16:52.076 --> 01:17:09.756
And so our collaborators were struggling to do that manually, and with AI we are now able to come up with very, uh, efficient designs, but also those we know actually work [laughs] because we have already the simulation as part of the loop saying that they work well.
435
01:17:09.816 --> 01:17:23.276
So I think these are examples where we see that it's not just about simulation, it's about really novel designs and novel discoveries that enable us to move the needle of innovation itself.
436
01:17:23.736 --> 01:17:26.876
Each one of these examples takes a lot of domain knowledge.
437
01:17:27.636 --> 01:17:41.276
How could somebody take your basic research if a domain expert and, uh, and quickly get started applying neural operators and the other frameworks that you've developed to their problem? Yeah.
438
01:17:41.776 --> 01:17:55.076
You know, neural operators are an open source library. It's extensively already adopted. It's part of the PyTorch ecosystem. It's, you know, used by a number of, uh, not only researchers, but also in companies.
439
01:17:55.396 --> 01:18:07.616
We have a lot of documentation there, so I encourage people to go there. We have like, you know, many different architectures, examples, recipes. So I think that's a great place to get started.
440
01:18:08.136 --> 01:18:17.776
You recently joined the UN Scientific Advisory Board. I know we're running out of time, but maybe just can you quickly give a bit of the story behind this and, uh, what you hope to accomplish? Yeah. Yeah.
441
01:18:18.176 --> 01:18:28.556
You know, I'm really honored to be part of the advisory board for the UN and in these tricky times with [laughs] a lot of geopolitics there, you know, which again, I'm not the expert on that.
442
01:18:28.616 --> 01:18:37.456
But when it comes to in-you know, aspects especially related to AI, having scientists in the room is something that, you know, I think is very important.
443
01:18:37.716 --> 01:18:52.196
I hope I could have an unbiased view and try to provide scientific evidence for any aspect, right? We wanna think about how AI impacts globally, like, you know, how do we ensure the benefits of AI reach everybody?
444
01:18:52.616 --> 01:19:04.576
How do we democratize access to AI? How do we ensure the unintended consequences and harmful impacts can be controlled? I think these are just the beginning aspects.
445
01:19:04.916 --> 01:19:15.056
Of course, the other side when it comes to weather models, I'm already excited, like, you know, there are, uh, there is a push to seeing how we can have better weather and climate modeling, so...
446
01:19:15.316 --> 01:19:19.096
then our food, you know, like using weather for better agriculture.
447
01:19:19.496 --> 01:19:31.116
So all these aspects are also where UN has a lot of dedicated agencies and people on the ground across the world, so I'm looking forward to contributing and being part of this.
448
01:19:31.636 --> 01:19:39.276
You know, looking, you know, looking at your career and how you, you know, talk and what you work on, it seems like you very much are a person who likes to solve concrete problems.
449
01:19:39.776 --> 01:19:46.696
You don't like to philosophize about things which... And you also seem, I think, maybe more optimistic than a lot of people in the AI space.
450
01:19:47.176 --> 01:20:02.616
Um, you have a very hope- like hopeful view of the world, um, I think not honestly always true. Um, what are the ways that you can uniquely bring that viewpoint to the board versus maybe some, you know- Yeah... living?
451
01:20:02.676 --> 01:20:08.136
Thank you. I, I... You know, to me, I think, as I said, I try to be unbiased and as a scientist- Mm-hmm...
452
01:20:08.236 --> 01:20:19.116
and, uh, as a scientist, I, I think that there's a lot of beneficial aspects of AI that are sometimes missed when we think of only the harmful impacts, right?
453
01:20:19.176 --> 01:20:36.636
And, and especially that is with respect to AI for science because a lot of regulatory frameworks equate AI with language models, and yes, language models can, you know, have manipulate people, can have all these kinds of harmful impacts that we should think about controlling.
454
01:20:37.136 --> 01:21:05.552
But AI for science is different, so I think this one size fits all is where a lot of- Problems come up, so we have to be mindful that there is, you know, AI that can change the world with new discoveries, and we should enable people around the world to not only benefit from them, but also be able to do research, you know, have access to AI that they can go innovate and use them in, uh, interesting ways.
455
01:21:06.132 --> 01:21:20.012
One question that we have been trying to ask every guest is, if you could pick a bottleneck in your domain that you could magically remove, what would that be and why? More compute. [laughing] Okay.
456
01:21:20.032 --> 01:21:24.592
[laughing] You know, that's the... I know that's an easy one or maybe a lazy one, right? Yeah.
457
01:21:24.612 --> 01:21:36.892
Because, you know, and, and you know, of course our compute that we have is growing so much more than even a few years ago, thanks to NVIDIA, thanks to others. Again- You may be biased there. Yeah.
458
01:21:36.912 --> 01:21:46.252
[laughing] So no, no comments on that. But what I mean by that is also like for research, enabling more and more compute, you know, it's very important.
459
01:21:46.392 --> 01:21:58.812
I know there are national labs building more com- supercomputers, uh, you know, hoping that we can have more compute for research. But I think, you know, without that we cannot experiment, we cannot innovate.
460
01:21:58.912 --> 01:22:06.672
I think this is a part that I push a lot and [laughs] you know, I think I cannot emphasize that it's so critical.
461
01:22:07.372 --> 01:22:17.752
If you had a call to action or a, um, something that you would like people to do or think about or, or learn about, what would that be?
462
01:22:18.352 --> 01:22:27.972
Yeah, so, you know, can go to Neural Operator Library, so you can kind of hands-on play with different architectures, recipes, you know, look at use cases.
463
01:22:28.512 --> 01:22:40.992
Uh, but also think about like, you know, AI for science as not just language models and agents. Yes, that's one aspect of it, but ultimately, you know, those are still like external wrappers in a way, right?
464
01:22:41.072 --> 01:22:54.732
Until we have AI that fully understands the physical world, not just as symbols, but as one that can simulate and design and control based on that, you know, there's a big piece missing.
465
01:22:54.832 --> 01:23:09.072
So that's the other aspect that I think the people should really think about AI for the physical world in this way. Anima, this has been so fascinating. Um, I, I'm excited to check out Neural Operators myself.
466
01:23:09.152 --> 01:23:17.532
I have some ideas in my head already. I really appreciate you taking the time to sit down with us. Thank you, RJ. It's been great. Thank you, Brandon. I really enjoyed it.
467
01:23:17.652 --> 01:23:31.392
We really dug deep into [laughs] a number of things. [laughs] So I appreciate you doing that. Thank you. Okay. Thank you. [outro jingle]
