🔬 The Coolest Diffusion Research Isn't in LLMs — Evan Feinberg & Sergey Edunov, Genesis Molecular AI

WEBVTT
1
00:00:00.180 --> 00:00:13.540
I remember very clearly in like twenty seventeen, twenty eighteen talking about GANs and how generative adversarial networks and how, uh, they're clearly the future of image generation obviously.
2
00:00:14.080 --> 00:00:28.360
They didn't work very well for, for proteins or protein-ligand systems, and we sort of had to wait for the right primitive to get created, and that turned out to be diffusion, which turned out to be a much more useful primitive for the space.
3
00:00:28.620 --> 00:00:39.880
What's kinda cool is right now for people that are, are interested in really core fundamental AI research, actually some of the most innovative diffusion research is happening in our field.
4
00:00:40.140 --> 00:00:52.380
It's happening in three-D structure prediction right now. No one would've predicted that then, but now, like, that's a pillar of, of diffusion, I'd say. Hi there. Welcome to the Latent Space AI for Science podcast.
5
00:00:52.760 --> 00:00:55.160
I'm RJ Haneke, CTO of Mira Omics.
6
00:00:55.340 --> 00:01:12.400
I'm joined by my co-host, Brandon Anderson, and we're privileged to have Evan Feinberg, founder and CEO of Genesis Molecular AI, and Sergey Ulanov, who led Llama two and Llama three pre-training before he joined Genesis as CTO.
7
00:01:12.860 --> 00:01:22.940
Hi. I'm Sergey. I studied physics at school. It was a long time ago. Um, and after graduation, I happened to work in software engineering. Thought I will never need physics again.
8
00:01:22.960 --> 00:01:31.760
[laughing] But then ML came up and became a thing, and turns out that a lot of things you're doing in ML were actually very similar to what you would do in physics.
9
00:01:32.320 --> 00:01:46.080
So I jumped in on ML bandwagon, and I did a lot of AI research, being a part of fair team in, in Facebook, uh, for quite a while. I later on led Llama team, um, Llama two and Llama three models.
10
00:01:46.860 --> 00:02:01.740
And then recently I decided to pivot my career again and, um, recover my roots in physics a little bit, and I joined Genesis as the CTO. Hey, I'm Evan. I'm the founder and CEO of Genesis Molecular AI.
11
00:02:02.300 --> 00:02:08.620
And, uh, like Sergey, also, also a physics major. I was a bit different from everyone in my family growing up.
12
00:02:08.680 --> 00:02:19.160
Almost everyone was a medical professional of some kind, and, uh, my sister became an accomplished TV writer, playwright, novelist, and my love was physics and computer science.
13
00:02:19.780 --> 00:02:28.000
Uh, but my mental model of an adult was, "Well, you should help people and help patients ideally." So, um, I was always searching for the right, the right way to do that.
14
00:02:28.300 --> 00:02:43.460
And, um, after arriving at Stanford doing my PhD in Vijay Pande's lab, we were excited in the mid-twenty tens of everything amazing going on in machine learning, to use a dated term, uh, for, for images and, and for language.
15
00:02:44.040 --> 00:02:57.200
Um, and around the same time that, that Sergey was at FAIR working on a lot of big graphs, I was at Stanford working on many small graphs, 'cause molecules are really networks of atoms and bonds and spatial interactions.
16
00:02:57.840 --> 00:03:11.420
And we were at the right place and the right time to bring to bear our backgrounds in, in physics to improve AI algorithms for looking at molecules. Uh, we published a few papers in the area of graph machine learning.
17
00:03:11.740 --> 00:03:15.720
And sort of like Sergey, I thought, "Well, I won't need this physics again," 'cause, you know, there's machine learning.
18
00:03:15.800 --> 00:03:33.560
But turns out the massive amounts of simulations on GPUs of proteins that we ran also came in quite, quite handy as Genesis evolved, and we've been really excited the past few years to figure out how to build foundation models for a totally new domain and make them useful for patients.
19
00:03:34.000 --> 00:03:43.880
Yeah, that's a really nice lead into my first question, which is, I, you-- we've both been in this domain of sort of machine learning for s- molecules and bio for roughly ten years.
20
00:03:43.940 --> 00:03:50.080
It's kind of a entire generation of tech bio has- [chuckles]... you know, come and gone since then. Yeah.
21
00:03:50.480 --> 00:03:59.400
While a lot of machine learning for molecules has been, I think, quite effective, one domain where it's been not effective has or historically really resisted, uh,
22
00:04:00.340 --> 00:04:03.580
machine learning modeling has been the world of protein small molecule interactions.
23
00:04:04.240 --> 00:04:14.580
And with some of recent advances that Genesis has put out, it seems like you might have actually started to make real improvement on this in a way that we haven't seen for a long time.
24
00:04:15.000 --> 00:04:30.160
Can you talk about what you have done and what Genesis has done, the developments which led to improvement, and why you think this is actually a real improvement over kind of some of the traditional, uh, machine learning strategies, which were ambiguously helpful?
25
00:04:30.880 --> 00:04:36.760
I totally agree, Brandon. It's-- And the amazing thing is, or, or one of the, one of the really remarkable things
26
00:04:37.740 --> 00:04:48.000
is that when we were founding the company initially as a, a spin-out of the research we were doing in AI at Stanford, um, we were afraid that we could be too late.
27
00:04:48.020 --> 00:04:59.780
[laughing] And, uh, I mean, you remember that, well, you and I first met around that time, right? Just seven plus years ago. As you rightly point out, almost a different generation in the area.
28
00:05:00.300 --> 00:05:09.160
And at the time, we were a, a little seed company. You know, there were already incumbents in the space who had raised orders of magnitude more, more money than us.
29
00:05:09.740 --> 00:05:20.340
There were already some really exciting academic papers that were out, and we had to constantly answer the question sort of like, "Is there room for another company doing AI in drug discovery?"
30
00:05:20.360 --> 00:05:28.700
[chuckles] Wh-which is a crazy thing to say, uh, i- it, looking with, with a backward-facing light. And, you know, hindsight is twenty/twenty, I suppose.
31
00:05:28.740 --> 00:05:40.500
And the claim that we made then is the same that we made, we're making now, which is that there's twenty thousand protein coding genes, and each of them can, can cause a disease.
32
00:05:41.200 --> 00:05:52.180
And in the same way that in other domains of AI, it's not like they were zero to one and then suddenly that problem was solved. The reality has been there's been no single iPhone moment.
33
00:05:52.820 --> 00:05:57.140
Um, even the iPhone required iterative development over time to become what it is today.
34
00:05:57.900 --> 00:06:07.936
And people love to talk retrospectively about the ChatGPT moment, but- The first ChatGPT models that became widely used were vastly less useful than, than they are now.
35
00:06:08.456 --> 00:06:21.656
And I think we should look at drug discovery from an AI context in the same way that as we go, we should expect to solve more and more problems, and there will be large epiphanic leaps.
36
00:06:21.896 --> 00:06:30.436
There will be iterative improvements, but they'll compound over time. And that's why I think in the same way that ten years ago was very early, but
37
00:06:31.416 --> 00:06:36.156
clearly value was starting to be created for drug development then from a, from an AI perspective.
38
00:06:36.936 --> 00:06:44.196
Fast-forward ten years, our technologies, which we'll talk about [chuckles] in this discussion, are vastly more useful than they were then.
39
00:06:44.776 --> 00:06:54.936
In the next ten years, we'll see another exponential improvement as well, in the same way that your car, in terms of autonomy, can do things today. We're, we're recording this in San Francisco.
40
00:06:55.396 --> 00:07:05.756
You look outside and, and see, see Waymos. We'd be blown away, you know, a few years ago by, by those technologies. So I think we should both be looking at it as
41
00:07:06.816 --> 00:07:17.916
every few months, every year, the technology gets more and more useful for discovering new medicines with artificial intelligence, and that will continue to, to be true for years to come.
42
00:07:18.596 --> 00:07:28.076
Maybe we can go back in time a little bit and explain, like, what was the state of protein small molecule drug discovery, let's say, about a decade ago?
43
00:07:28.456 --> 00:07:39.496
Like, what could we actually expect to do with machine learning models, and what were some of the failure modes that people, I think, didn't really see coming from a practical standpoint back then? It's funny.
44
00:07:39.636 --> 00:07:48.936
Just, I mean, breaking the fourth wall a little bit, I, I, I came in not, not really knowing any of these questions, but Brandon really is knowing how to speak my love language [laughs]
45
00:07:49.176 --> 00:08:06.256
Because, because like, like the other people in this room, you know, though my, my background is much more into the physics and CS side, I've been very, very focused in a deeply passionate way about the area of how can we make medicines with AI for over a decade.
46
00:08:06.316 --> 00:08:18.156
So I've seen so, so many tectonic shifts a-as of view. To give you one concrete example, and, and I'll give a little background for those who don't have as much biological background.
47
00:08:19.196 --> 00:08:29.736
Drug discovery is akin to finding a key for a lock, where the lock is usually a protein, in some cases a nucleic acid. [laughs] Right?
48
00:08:30.376 --> 00:08:44.496
Um, and where the drug i-is the key, and it's usually some, some small molecule or a peptide or, or, or an antibody or some other modality that wants to bind that lock, that protein, the receptor, and change something about it.
49
00:08:45.156 --> 00:08:51.436
If it's an enzyme, you wanna usually stop it from functioning. Sometimes you want to activate that endogenous protein.
50
00:08:51.976 --> 00:08:58.376
But it's to, it's to introduce an external molecule, an exogenous molecule to change something about the biological pathway.
51
00:08:59.336 --> 00:09:07.856
A necessary but not sufficient part of that process is finding a molecule that binds well to that receptor or that protein. Of course, that's not sufficient.
52
00:09:07.956 --> 00:09:15.156
We also need to make sure it does not bind to certain anti-targets, certain proteins you want to avoid, that often mediates toxicity.
53
00:09:15.796 --> 00:09:24.796
You need ADME and toxicity, which is typically like thirtiest properties to make sure your molecule is safe in addition to being effective and gets to the right tissue.
54
00:09:25.536 --> 00:09:34.756
Uh, these are all critical, and all are necessary and none are sufficient. But one concrete example to answer your question about protein ligand interactions
55
00:09:35.816 --> 00:09:48.056
is that we had a hypothesis for a long time that if we can predict the three-D structure, the three-D coordinates, imagine a, for the CS people, like a point cloud,
56
00:09:49.056 --> 00:10:03.176
um, the three-D positions of your drug and the three-D positions of the protein, and if you could model that with high accuracy, that will lead naturally to measuring binding affinity or pr- or predicting potency more accurately.
57
00:10:03.816 --> 00:10:14.096
That was a hypothesis that fundamentally could not be tested because models for predicting those three-D poses, the three-D structures of complexes, were so bad.
58
00:10:14.356 --> 00:10:22.196
[laughs] Or if you can predict this, it requires so much computational tech resources that you might as well just solve the problem. Right.
59
00:10:22.275 --> 00:10:31.336
Like, you might as well solve the three-D crystal structure or cryo-EM structure, is what you're saying, which, which can cost tens of thousands of dollars. Um, it can be months or years. PhD thesis, yeah. Right.
60
00:10:31.376 --> 00:10:37.636
Entire PhD thesis, postdocs can sometimes work literally twenty-four seven trying to solve one three-D structure.
61
00:10:38.336 --> 00:10:58.276
And the hypothesis was that if we make this orders of magnitude faster, sometimes more accurate, which we can talk about, with artificial intelligence or before that with, with molecular dynamics, that you would thereby not only accelerate drug discovery, but enable the discovery of medicines for targets that were previously thought to be undruggable.
62
00:10:58.956 --> 00:11:05.816
And that was just a hypothesis, and it, and it took until the last few years to show that if we systematically improve
63
00:11:06.896 --> 00:11:16.216
those, the accuracy of those, those predictions, if we can improve the accuracy of a protein ligand three-D complex, we can make potency prediction more accurate as well.
64
00:11:16.776 --> 00:11:27.236
And that's something that I am personally extremely gratified to feel like we've been showing because that was just a hypothesis when we started the company a few years ago.
65
00:11:27.336 --> 00:11:40.976
And, and now it's become a reality thanks to the convergence of a lot of different ideas that we're able to, to, to enable Pearl and other foundation models like it. Yeah. Can you talk about Pearl a bit?
66
00:11:41.216 --> 00:11:49.616
Um, so the-- I think one of the interesting things about Pearl, in my mind, is it was an attempt to kind of scale what seems unscalable to me.
67
00:11:50.116 --> 00:11:55.576
Uh, RNA small molecules, you know, normally if you scale them, models just come out as just, like, pure pattern matchers.
68
00:11:55.596 --> 00:12:03.116
[laughs] These things love to tell you what you already know, and sometimes they're maybe not so great at telling you something you don't know.
69
00:12:03.788 --> 00:12:12.248
So maybe it seems like there's hints that Pearl has actually moved beyond this. Um, like what were the key insights which went into this? Sure. And by the way, Brandon, like what you described- Mm-hmm...
70
00:12:12.728 --> 00:12:22.688
that shouldn't surprise any, like, pure ML practitioners in the audience just because machine learning is initially constructed for pattern recognition.
71
00:12:22.728 --> 00:12:31.248
And it's the same with when you fire up Claude or ChatGPT or what have you. It's going to do best when it's closest to the training data, and it's going to pattern match off it. Yeah.
72
00:12:31.668 --> 00:12:45.388
So I think that has been the big sense of urgency in AI meeting the physical world is how to extrapolate, how to make generalizable models, and that's, that's we've been really hard at work on.
73
00:12:45.728 --> 00:12:51.888
Maybe, maybe Sergey wants to give some more, more color on that. Maybe to step back a little bit and discuss what Pearl is.
74
00:12:52.068 --> 00:13:08.847
It's a structure prediction model, which basically means it takes as an input a protein sequence and a ligand representation, what you try to attach to this protein, and it predicts how this protein and ligand are gonna look together, uh, as a structure in the 3D space.
75
00:13:09.288 --> 00:13:16.208
So like a term for this people sometimes use is like co-folding. Is that- It is called co-folding, right. Yeah. And AlphaFold3 and then some of the...
76
00:13:16.348 --> 00:13:26.808
and then BOLTS and, um, OpenFold3 and so on have also implemented some of these ideas in their own way. Um- Yes. Yeah. There are, there are similar models. Some are open source, some are closed source- Mm-hmm...
77
00:13:26.828 --> 00:13:38.848
but are pursuing- Yeah... similar, similar direction. Our fundamental difference is that we are focusing on small molecular space. Uh, there are a lot of models that are doing protein-protein interaction, interactions.
78
00:13:39.568 --> 00:13:51.488
Mm-hmm. And it works well. For small molecular space, on the first glance, it may sound like it's an easier task because, like, well, it's a small molecule in them, right? So, like, why would it be challenging?
79
00:13:52.148 --> 00:14:03.828
But the reality is the search space is so vast for, for small molecules. There are ten to the sixty drug-like small molecules in the universe. Uh, so good luck searching for the space.
80
00:14:03.848 --> 00:14:11.168
[laughs] And there are also all of, all variations, like you can rotate it. Uh, you can have different confirmation of those molecules.
81
00:14:11.188 --> 00:14:19.628
So, uh- Maybe the intuition there is that you have, like when you type in a Google search, if you just do a few words, you're gonna get a huge list of matches.
82
00:14:19.968 --> 00:14:28.128
But if you have a very specific query, then you're going to be-- you know, you're gonna have a few matches, and they're more likely to match.
83
00:14:28.168 --> 00:14:38.768
So similarly, with a small molecule, if you have a very small query, which is your molecule, then it's likely that you're gonna have to search a, a huge space of, of possible matches.
84
00:14:38.868 --> 00:14:48.588
Whereas if you have something very complex, then there's-- it's easy to rule out a, a, a match at that. Is that a good way to think about it? That's one way to think about it, definitely.
85
00:14:49.428 --> 00:15:04.968
Uh, for me, it is just the, the computational complexity on figuring out which small molecule will attach to this protein. Mm-hmm. It's such a vast, um, problem to solve that is im-impossible to do without good models.
86
00:15:05.068 --> 00:15:14.868
Mm-hmm. It's like finding a needle in a haystack- Yeah... where everything except your needle is very, very dangerous or, or ju- [laughs]... or just doesn't bind. Or they are needles after all. Right. Yeah. Yeah. Right.
87
00:15:14.948 --> 00:15:23.348
We're finding hay in a needle stack. Yeah. It might be a, a more accurate analogy. Oh, yeah. Yeah. Totally different. Yeah. Yeah. So, so now, now we can go and think about how we train those models, right?
88
00:15:23.488 --> 00:15:33.628
So a lot of training data that people are using- Mm-hmm... is so-called PDB, uh, which is a public database of like all of the historical crystal structures. And it's not that big.
89
00:15:33.988 --> 00:15:43.868
It's like two hundred thousand crystal structures, and it's very hard to expand. It takes a lot of, uh, time and energy and money, uh, to create new crystal structures.
90
00:15:44.148 --> 00:15:56.308
Although there are some projects that are pursuing this, it's still expanding at a glacial pace. Um, so expanding this database is pretty hard. But what we figured out is possible to do, and it's
91
00:15:57.248 --> 00:16:04.728
very relevant to our small molecular space, is that in small molecular space, you can actually model your small molecules with physics.
92
00:16:05.408 --> 00:16:15.108
Uh, you can model the-their behavior, and that allows you to create more data that you can train model on. Mm. Something which is not necessarily possible in protein to protein, um, I'd say.
93
00:16:15.128 --> 00:16:24.608
Because they're too complex to-- for a protein. Yeah. Those are very big molecules. Yeah. So it's very hard to model them with physics. Not impossible, it's just computationally very, very hard. Right.
94
00:16:24.828 --> 00:16:32.428
So, so I guess you-- what you're saying is you do a really good job of modeling these small molecules using ND or something like that.
95
00:16:32.888 --> 00:16:41.648
And so you can kind of put together those structures in your training set at a much lower cost, and so that you can use that as your synthetic training set.
96
00:16:42.048 --> 00:16:54.828
May-maybe to step back a little bit about our-- and talk about our roadmap. Um, so it's not fundamentally different from the roadmap for LLM scaling. Like, remember in LLMs, we have all of the stages.
97
00:16:54.888 --> 00:17:05.668
You have pre-training scaling, then you have post-training scaling, where you do, like, either fine-tuning or RL. Mm-hmm. Now everybody's doing RL. And then you do inference time scaling.
98
00:17:05.828 --> 00:17:14.508
So all of those three concepts, they connect. Yeah. And that, that's what led us to state-of-art LLMs these days. It's not fundamentally different on our side.
99
00:17:14.888 --> 00:17:28.908
Uh, we also have pre-training scaling, where we create a lot of synthetic data to train better models. Um, we do that. Um, then the second thing we have-- we started doing is actually the third step in LLM scaling.
100
00:17:29.388 --> 00:17:34.248
We started doing inference time scaling. Um, it's fundamentally very similar.
101
00:17:34.488 --> 00:17:46.728
Like in LLM, when we talk about inference time scaling, we're talking about thinking tokens, where LLM, instead of giving you the answer right away, it kind of goes and thinks for a while and then comes up with response, right?
102
00:17:47.468 --> 00:17:57.188
Well, we are doing very similar thing with our models, where a model is forced to think, except it's not thinking in language tokens, it's thinking in terms of crystal structures.
103
00:17:57.608 --> 00:18:03.528
Not fully materialized crystal structures, but some sort of a crystal structure representation memory.
104
00:18:04.360 --> 00:18:13.220
Uh, and model kind of goes back and forth with those, um, and we use physics-based guidance during this process to steer the model output in the right direction.
105
00:18:13.900 --> 00:18:25.280
And what we found is that it improves model performance by a lot. It's easy to understand how thinking tokens work because it's just basically parts of the transcript that you don't see.
106
00:18:25.800 --> 00:18:34.920
But I think that at least for other models, you have like sort of some sort of loop in... I-is that similar to what you're doing, like a loop transformer or something like that?
107
00:18:35.040 --> 00:18:45.840
Or is there, there's m-- I, I know you mentioned that there's some physics-based verification as part of that in the loop. I-is there more to it than that? One fundamental block of our models is diffusion-based head.
108
00:18:46.040 --> 00:18:58.400
Mm-hmm. So it's like the same diffusion models, what people are using for image and video generation. Mm-hmm. Well, we use them for crystal structure generation, right? And diffusion head is by nature iterative. Mm-hmm.
109
00:18:58.400 --> 00:19:08.479
It's like multiple steps. Uh, you're refining your predicted structure. Um, and as you're doing this process, you can steer the model in the right direction.
110
00:19:09.140 --> 00:19:15.680
So there's like, uh, you have like a steering simulation or something that in the, in the loop in that.
111
00:19:15.740 --> 00:19:28.140
So you, you go through a passive diffusion, you look at the, what comes out, you run some verifier and use that to say, "I like this" or "I don't like this" or give a vector to go towards. Is that kinda the idea? Yeah.
112
00:19:28.560 --> 00:19:37.220
So, so it's like your diffusion head is basically learning something like a force field, and that you're kind of bouncing off between a diffusion-based force field and like a physics-based force field?
113
00:19:37.940 --> 00:19:50.040
Or is that a way of thinking about it or? I struggle to understand like what those models are really learning underneath. Mm-hmm. And I think the whole idea of explainability is a big research challenge. Mm-hmm.
114
00:19:50.060 --> 00:20:00.220
Yeah, you can identify what a neuron and a transformer potentially thinking about, but it's really, really difficult problem. So we, we have some internal representation- Mm-hmm...
115
00:20:00.260 --> 00:20:06.560
but at the output comes out as a crystal structure, but it's really hard to tell what exactly is happening inside.
116
00:20:07.020 --> 00:20:19.080
Is it possible to do something to the effect of the standard me-mechanistic interpretability, things like sparse autoencoders and the more complicated things in this context, or is there a problem with doing that kind of thing?
117
00:20:19.580 --> 00:20:26.920
I think it would be interesting research, as a research direction. That's not something we are pursuing. I guess potentially we could explore this. Oh, yeah. Yeah.
118
00:20:27.520 --> 00:20:34.920
Sergey's a lot more credible than I am talking about the [laughing] Go for it. LLM translatability to aerospace.
119
00:20:35.540 --> 00:20:47.620
Sergey is being humble, but S-Sergey led the Llama Two research team at, at, at Meta when, when he, when he was still there, so he's trained one of the most widely used language models in the planet.
120
00:20:48.060 --> 00:20:59.960
But I guess what I can add on the, on, on more the, the physical interpretability side is I like to look at, at AI models in terms of the inputs, the model themselves, and, and the output.
121
00:21:00.320 --> 00:21:18.520
And, um, we find it really improves outcomes as well as interpretability to your question, when you could include as many physical priors as possible without, of course, you know, biasing the model to what we puny humans believe.
122
00:21:18.980 --> 00:21:23.700
But I've always had the view that AI fundamentally is representation learning.
123
00:21:24.480 --> 00:21:39.640
And to me, that's actually not that different than what's happened in the language and the vision domains, where we're enforcing a real human prior on images when we start using convolutional neural nets, is we're saying, well, pictures are grids of pixels.
124
00:21:39.960 --> 00:21:53.280
There's nothing inherent about that. That's, that's a human construct. And we constructed ConvNets on top of that. And language as sequences of tokens in one dimension. We first built RNNs.
125
00:21:53.340 --> 00:22:05.060
We've then built transformers, but it's still, that's baking in a pretty, pretty serious prior on how to, to look at that data. And we don't view it very differently here. In our case, from the
126
00:22:06.040 --> 00:22:17.100
input perspective, the disadvantage that we have in our field over the more traditional demands of AI is that we don't have the internet to work with.
127
00:22:17.160 --> 00:22:27.480
Like, we can't just download Reddit posts and, you know, buy some subscriptions to the Wall Street Journal and, like, train a model, and voila, the pre-trained model works fairly well.
128
00:22:27.920 --> 00:22:37.620
The c-closest equivalent that Sergey mentioned is the RCSB Protein Data Bank, which has more like a couple hundred thousand crystal structures, prior structures and others, although there's a lot more...
129
00:22:38.220 --> 00:22:42.760
there's a lot of tokens per structure, so there's a lot more latent information that that figure would describe.
130
00:22:43.600 --> 00:22:51.840
So we have to be clever on the input side, which is generating more pre-training data, as, as Sergey's mentioning, um, using physics as much as possible.
131
00:22:52.480 --> 00:23:01.700
In terms of the model architecture itself, the idea is to not have the model have to relearn as little physics as possible, so is less likely to overfit.
132
00:23:02.260 --> 00:23:15.620
And then at the output stage, um, enforcing, enforcing physicality, I think that also goes to one of our main, uh, focuses as a company, which is we've been focused from the beginning on small and medium-sized molecule discovery.
133
00:23:15.720 --> 00:23:24.200
So what does that mean? So small molecules tend to be, uh, drugs that you can take as a pill, so it's orally bioavailable. It's how most people think of medicine.
134
00:23:24.780 --> 00:23:34.600
And medium-sized molecules, so think about macrocycles, peptides, um, modalities that break the traditional rule of five but are still growing, uh, growing modalities in medicine.
135
00:23:35.380 --> 00:23:41.500
That said, of course, small molecules are still sixty-five percent of FDA-approved drugs, so we're talking about the biggest part of the pie.
136
00:23:42.040 --> 00:23:57.180
We're building this with the judicious focus from day one on what do drug hunters, what do medicinal chemists, what do CAD scientists, what do they need to dis-- to discover drugs faster and better than they could before?
137
00:23:57.660 --> 00:24:07.840
And so we wanted to, to build that sort of, um, human, not only interpretability, but usability from the beginning, and also physical usability.
138
00:24:08.600 --> 00:24:19.020
We want outputs from these models to be used by ph- Physical methods that require 3D coordinates to actually make sense. You use the term force field.
139
00:24:19.520 --> 00:24:26.560
We want the outputs of our model to be useful as inputs to a force field. There is this, uh, a few papers that came out.
140
00:24:26.720 --> 00:24:36.680
One was in Cell, I think, last year, which showed that for all the claims about AlphaFold solving drug discovery, people try to take AlphaFold produce protein structures,
141
00:24:37.980 --> 00:24:49.780
use them for traditional docking, and found no value in it. Those pockets just weren't high resolution enough. They couldn't be used by a physical screening method.
142
00:24:50.360 --> 00:25:05.780
And so we wanted to be able to build our systems to be useful by humans, useful by all the many adjacent and powerful tools from the physics and computational chemistry communities from day one for-- to have that interoperability.
143
00:25:06.100 --> 00:25:09.820
Make sense? Yeah. So, uh, this might be a good time to just step back for a minute.
144
00:25:10.180 --> 00:25:26.200
When you develop a drug, there are many steps to that, from discovery and toxicity and availability, and so there's names for all these things which I'll let you, you state, and then there's the clinical stuff and finally, uh, eventually approval, hopefully.
145
00:25:26.720 --> 00:25:37.740
Where do, do your models sit in that? It-- Or which parts of that process do they sit? And what, what are, what are the steps? Just, like, briefly, I've read that there's twelve steps, maybe.
146
00:25:37.920 --> 00:25:48.040
[laughing] I don't know the twelve steps in drug discovery. Acceptance is the last one. [laughing] Acceptance that your phase th- phase three trial failed. Yeah. Exactly, yeah.
147
00:25:48.520 --> 00:25:59.280
So where do, where do you-- do your models help people to do their jobs in the discovery or in the development process? As you rightly point out, if we go from, from the very beginning to the very end,
148
00:26:00.260 --> 00:26:09.500
first we need to identify what target is causing the disease, what signaling cascade, what's going wrong, what's causing the phenotype of the patient.
149
00:26:10.200 --> 00:26:21.020
And that's not a trivial exercise, but, um, there are many, certainly not all, but many diseases where the cause is actually very clear.
150
00:26:21.440 --> 00:26:34.380
The most clear are the, the monogenetic diseases, but even in others, it's from various panels, it's clear that this overexpressed or overactive or deleted gene or protein is causing the disease.
151
00:26:34.580 --> 00:26:51.000
So we know many, but not all of the universe of causes of diseases. Then once a target is identified, we have the process of finding a drug for that target that's ideally selective, very potent.
152
00:26:51.540 --> 00:27:00.200
Um, it gets the right tissue. Some diseases are multi-system. Some are very specific to, to a certain organ or tissue. Wanna make sure the drug can get that tissue.
153
00:27:01.120 --> 00:27:12.960
And then there is the process of GLP tox and IND enablement, IND, investigational new drug. You have- So GLP in this case is, uh, not glue. Not the [laughing] Not, not- Yes.
154
00:27:13.000 --> 00:27:24.240
[laughing] We're, we're not, yeah, we're not- Good lab practices. Yeah. Yeah. Thank you. We're not talking about the metabolic space right now. [laughing] Um, there's, there's clinical trials, usually ascending size.
155
00:27:24.280 --> 00:27:35.060
Mm-hmm. Your phase one starts typically more, more with safety, um, then phase two and phase three, and, and as you point out, ideally approval at whatever regulatory body, the FDA or NDA or, or what have you.
156
00:27:35.700 --> 00:27:44.860
Um, our contention is that the highest leverage application of artificial intelligence is the drug discovery and drug design process.
157
00:27:46.020 --> 00:27:56.340
And the reasons are that even though they're, they're all valuable, and I cheer on all the peers, many of whom I know, who are working at all the different parts of this stack.
158
00:27:57.020 --> 00:28:03.960
But I think the first thing to make clear is just like a, a vision model is going to be very different than a coding model.
159
00:28:04.360 --> 00:28:18.500
They might share some similarities under the hood but requires real focus to do any one area correctly. Um, there's a similar, if not greater difference from the sort of models you need for target identification, i.e.
160
00:28:18.520 --> 00:28:31.060
biology to drug discovery, to preparing for regulatory filings, to helping segment patients for clinical trials. These are all very important complementary, but ultimately distinct problems.
161
00:28:31.800 --> 00:28:46.020
Um, I think about all of the times where a patient goes to a doctor, gets t-told that they have a very clear diagnosis, and in addition,
162
00:28:46.980 --> 00:28:55.960
the physician will tell them there's been breakthroughs where we understand why this disease happens. We've sequenced your genome even. We, we know that-- or sequenced your tumor.
163
00:28:56.360 --> 00:29:02.540
We know it's causing your condition, but we do not have a selective therapy. There is no precision medicine for your condition.
164
00:29:03.260 --> 00:29:09.800
And it has the hope that still, at least they know what's wrong with, with me, but they don't really know how to treat it.
165
00:29:10.400 --> 00:29:20.280
And our aim is to, um, have as many moments as possible where patients are told they have a very specific condition, and there's a specific treatment for them.
166
00:29:20.860 --> 00:29:40.040
That's going to require bending the cost curve of develop-- discovering developing new medicine, but also importantly, solving those cases where there's certain, quote, undruggable targets, undruggable proteins that we need to figure out how to drug, where those targets have, have proven, uh, resistant to traditional methods or even intractable in some cases.
167
00:29:40.780 --> 00:29:42.800
Just to clarify my understanding.
168
00:29:43.560 --> 00:30:04.520
There's the discovery part, and then there's the clinical part where-- And, and so there's, there can be effort there where, okay, I wanna find what's the target that is impacting this disease the most, or how do I maybe what's the thing that I want to go after in order to solve a medical problem.
169
00:30:04.640 --> 00:30:11.660
And then there's the clinic where you're answering, does this thing actually work? And maybe there's a feedback loop between them even.
170
00:30:12.000 --> 00:30:23.440
But if you don't have the thing in the middle that's like, "Okay, here's how we actually build this drug, it-- so that I can't hit my target, it's not selective enough, so it, like, kills cells or does things to cells that I don't want it to do."
171
00:30:24.000 --> 00:30:35.612
So then it doesn't matter, right? Like, I can have the answer- This drug, we should go after this target, but then if I don't know how to do that effectively, potently, et cetera, then, then the answer doesn't matter.
172
00:30:35.872 --> 00:30:44.192
Is that kind of what you're saying? Yeah. RJ, if you... S- 'cause people love to debate about success rates in, in, uh, clinical trials.
173
00:30:44.992 --> 00:31:11.572
But the reality is, if one focuses on those drug candidates that are aimed at proteins that have a close genetic linkage to a disease and/or at least where the biology is well understood, where the animal models translate well to the disease, where the molecule is predicted to have good pharmacokinetics, as in the levels in the blood in patients or in serum are gonna be high enough,
174
00:31:12.472 --> 00:31:22.912
those molecules have fairly high FDA approval rates. The success rate from fa- phase one to end of phase three is substantially higher than average.
175
00:31:23.472 --> 00:31:34.752
People love to cite the ten percent success rate, but that's really a low ball, um, because we often know what genes, what proteins, what targets are causing the disease.
176
00:31:34.812 --> 00:31:43.852
They're just really hard to drug, or the molecules we put into, in the patients are inferior to the, the, the target product profile that, that they deserve.
177
00:31:44.372 --> 00:31:59.452
In the cases where if you just look at the preclinical data, you focus on good targets with good biology that are predicted to be distributed well and have good safety profiles, those molecules are very likely to get approved and create tremendous value for patients.
178
00:32:00.032 --> 00:32:09.552
And so our view is, I think about the physicians that run trials that are clamoring for those kinds of molecules. I think about the patients that are looking for more selective therapies.
179
00:32:10.232 --> 00:32:20.272
And so our view is that is the highest leverage application of AI in healthcare medicine more broadly. Just a business question here.
180
00:32:20.832 --> 00:32:30.832
What is the space of things which are-- the, the target is the biology is understood, the structure is understood, and that, you know, the only thing you really need to do is find the right lock.
181
00:32:31.392 --> 00:32:44.332
That seems like you probably would have already picked over that class of targets. In some sense, those are easy targets, right? So how, how much is it opportunity is there for that? So there's two orthogonal concepts.
182
00:32:44.852 --> 00:32:59.632
Known biology is orthogonal to the ease with which one can drug that target. Sometimes, unfortunately, it seems they anti-correlate. [chuckles] In that often it seems that the most appealing targets from a,
183
00:33:00.692 --> 00:33:09.792
a, a validation perspective seem to be really hard to drug. Mm-hmm. Yeah. So, so do they anti-correlate, or is it just that we have picked all the low-hanging fruit which do solve both those?
184
00:33:10.052 --> 00:33:20.372
I, I, I think it is likely the case that many of the, quote, "easier targets"- Mm-hmm... for a variety of reasons have been s- sort of, many of them have been drugged, not all. Mm-hmm.
185
00:33:20.772 --> 00:33:32.332
But even in those cases, um, people love to, to sort of, um, have this, this false dichotomy of easy versus hard targets and, well, this one's drugged versus it's picked over, therefore it's not.
186
00:33:32.432 --> 00:33:38.552
But we at, at Genesis do most of our work through large pharma. Mm-hmm.
187
00:33:38.692 --> 00:33:52.512
Like, most of what we do is providing AI services basically to, to, to, to major pharma companies like, like Gilead, and we recently announced our, our, um, expansion of our, of our collaboration with Insights, which we're, we're excited about.
188
00:33:53.292 --> 00:34:05.372
And I can't give details of specific targets that they're working on, but what I can say is there, there's a real range from first-in-class chemical matter. What that means is we believe this target causes a disease.
189
00:34:05.572 --> 00:34:12.132
There's no known molecules that bind to that target. We need to find the first binders ever, the true zero to one cases. Mm-hmm.
190
00:34:13.092 --> 00:34:43.451
But there's a l-- um, a, a wide variety of targets we work on that are more, I'd say, one to ten cases where sometimes there's a-- there is an approved agent, or sometimes there, um, uh, there's molecules that are only preclinical, but they're not optimal, where if you can improve upon those preclinical molecules and get them to development candidates, so we're going to get new patients, or if you can improve upon the existing clinical or approved agents, you can create a lot of value for patients.
191
00:34:43.972 --> 00:34:45.432
I'll, I'll give an ex-- a public example.
192
00:34:45.532 --> 00:34:56.812
We're, we're not working on this target at all, but you just look at the, the progression of ALK inhibitors, ALK inhibitors, and you look at, to make it very concrete, charts of patient survival.
193
00:34:57.532 --> 00:35:03.252
You might have said, "Well, when the first ALK inhibitor came out, why do we need another? We've drugged ALK. We're done."
194
00:35:03.992 --> 00:35:20.232
But it is a qualitative, not just a quantitative difference when you, when you compare patient survival curves from first generation ALK inhibitors to later generation ALK inhibitors of how much longer those patients live, how many more get benefit from it.
195
00:35:20.852 --> 00:35:33.952
And so I think there's not only value in first in class, there's enormous value for patients in best in class too. And, and I think both are areas that, that we focus on, if, if that makes sense. Okay. Yeah.
196
00:35:34.012 --> 00:35:46.612
So we so far have talked about Genesis modeling in terms of, uh, let's say this one specific problem of protein ligand binding. How does your overall world work in terms of developing this?
197
00:35:46.732 --> 00:35:56.952
I mean, I assume that you must ha-- I mean, I, I do know that you have other things that you've worked on in solving the broader early phase drug discovery, um, problem.
198
00:35:57.752 --> 00:36:09.132
I keep joking about that when Nobel Prize was given for AlphaFold three, a lot of people thought that, like, drug discovery is solved. [laughs] And it's very far from the truth. No, very, very, very far.
199
00:36:09.372 --> 00:36:19.572
Um, like, yeah, you can maybe, maybe, so maybe you can predict crystal structure, right? At low resolution. At low resolution. Yeah. Well, we're, we're doing better, I think. Yes. Um,
200
00:36:20.552 --> 00:36:26.532
assume you can predict crystal structure. Does it mean the drug discovery is solved? No. No. Obviously not. Yeah.
201
00:36:26.572 --> 00:36:38.392
The broader community, like protein folding community and drug discovery community immediately said there's other things you care about, like dynamics, like just a sing- single static structure is not enough to understand what's going on, interactions, um-
202
00:36:39.428 --> 00:36:43.368
Yeah, so there's lots of things there in addition to just, like, having a static structure.
203
00:36:43.468 --> 00:36:51.528
I think many people thought those were very useful, like, they've really accelerated science, but it was very clearly a starting point and not a end condition. Right.
204
00:36:51.568 --> 00:37:01.408
But if you look outside of, like, people working in the field- Yeah... like general- Mm-hmm... popular community, it was a pretty wild, wide belief that- Mm-hmm... the problem is solved. Yeah.
205
00:37:01.428 --> 00:37:06.148
Now, in reality, besides crystal structure- Sorry, sorry. I just have to clarify that. Otherwise, like, there's an entire- [laughs]...
206
00:37:06.248 --> 00:37:15.288
ML structure biology community which will all just jump on us and say like- [laughs]... "No, no, not solved problem." Gotta, gotta throw those caveats in. [chuckles] Absolutely not a solved problem. Yeah. Yeah. Yeah.
207
00:37:15.328 --> 00:37:23.688
In reality, you need to predict so many other- Mm-hmm... properties. Yeah, yeah. Uh, like other properties that- Mm-hmm... I haven't mentioned. Like, basically, you're designing the key for a lock, uh- Mm-hmm...
208
00:37:23.708 --> 00:37:31.348
like that small molecule that sticks to your protein in the body. But you don't want the small molecule to stick to everything in your body- Mm-hmm... 'cause that's probably gonna cause issues.
209
00:37:32.068 --> 00:37:43.668
You also want to make sure that small molecule can be sol- like, is soluble, uh, so that you can ingest it as a pill. Uh, those are important properties, too. You want to make sure it doesn't have any other side effects.
210
00:37:44.588 --> 00:37:52.008
Um, and predicting all of those properties is also just as important as, or maybe even more important than predicting the crystal structure itself.
211
00:37:52.888 --> 00:38:00.548
Um, and of course, as a company, and we're very proud of it, um, we're not just building models for predicting crystal structure.
212
00:38:00.628 --> 00:38:12.608
We are building models for predicting all of those properties and basically enable drug hunters to kind of become 100X more effective, uh, in, in their daily job. I'm curious about this.
213
00:38:13.048 --> 00:38:20.148
The pipeline that you have, you mentioned you have different pharma partners, GSK or whomever you're working with.
214
00:38:20.668 --> 00:38:30.008
Do they have targets that were sort of drugs that were designed by you that are in, like, clinical trials, approved? Like, how-- where, where do we stand with all that?
215
00:38:31.068 --> 00:38:38.288
We're limited in what we can say about- Yeah, of course. Yes... these pharma companies are, are, are famously secretive and which makes sense. Yes.
216
00:38:38.528 --> 00:38:48.168
What I can say, whi- which was recently publicly disclosed as an example- Yeah... um, is, um, we just expanded our partnership with Insytes. Right.
217
00:38:48.228 --> 00:39:00.368
And that started approximately, uh, you know, a little over a year ago, the initial work together, and that spanned the two sort of bookends of the drug discovery process in some ways. So- Yeah...
218
00:39:00.788 --> 00:39:25.448
one of the, the, the programs we worked on together was a case where a very challenging target where chemical matter, um, existed, but there is a key binary event which is getting to a DC or a development candidate, and that is the molecule or ideally a set of molecules, all which are possible to be, um, uh, the agent for a phase one clinical trial.
219
00:39:26.108 --> 00:39:42.208
And, um, we had to, to do some work together where we, we take our foundational models, our base models, we fine-tune them on Insytes data and use them in, in a close collaboration to, to sort of work together to, to get to a DC faster.
220
00:39:42.708 --> 00:39:50.468
And we're getting, um, substantially closer [chuckles] in, in that case, which is, um, one of the reasons we're excited to, to expand our work together.
221
00:39:50.908 --> 00:39:57.828
And on the, the flip side of that is one of the other areas of work together was on a protein
222
00:39:58.728 --> 00:40:16.648
w-with also very nice linkage to a very severe disease where there was no known chemical matter, like no patents, no papers, in this case, no co-crystal structure of a ligand, another synonym for a small molecule, uh, that bound to that protein.
223
00:40:16.728 --> 00:40:23.408
So we had to find the first ever known chemical matter to bind to it, and then we'd progress those, what are called hits.
224
00:40:23.468 --> 00:40:40.688
Hits are like the first molecules that, that bind your, to your protein, uh, progress those in-into, um, inhibitors that are active in called biochemical assays, which are more enzyme-based than cellular assays, which means that in a actual cell, living cell model, um, your molecule is active as well.
225
00:40:41.028 --> 00:40:50.848
So it was really based on, on, on a concrete set of accomplishments together that we want to ex-expand the collaboration. And unfortunately, outside of that, [chuckles] there's really limited- Yeah, sure...
226
00:40:50.848 --> 00:40:52.328
that, uh, uh, in regards to what we can say. Yeah.
227
00:40:52.368 --> 00:41:17.168
But, um, the objective of Genesis is to create medicines that patients wish they had, and the way we'll be able to do that is by working with as many pharma and biotech partners as possible for whom their comparative advantage is discovery, clinical development, commercialization.
228
00:41:17.728 --> 00:41:30.728
And our comparative advantage, of course, is in AI where, where we can put those, those two expertises together in, in a synergistic, not just an additive way, and, and make medicines together that otherwise would, would not be possible.
229
00:41:30.928 --> 00:41:41.608
Yeah. Um, so at-- that's the name of the game that we're in this for is those, those clinical outcomes that you're pointing out, and I am very excited to be able to share those, um, as, as they arise in the coming years.
230
00:41:42.168 --> 00:41:56.048
So one thing that, uh, is-- you guys talk about in your website and whatever is this one Angstrom threshold at which a protein structure prediction and the binding between that and a small molecule becomes useful.
231
00:41:56.488 --> 00:42:03.308
Can you talk a little bit about, like, okay, why is that the case? What-- how did you do it when others have failed?
232
00:42:03.788 --> 00:42:11.268
And, you know, and you've spoken a little bit about that w-with the, you know, sort of biases in the model, the synthetic data, maybe there's more.
233
00:42:12.128 --> 00:42:21.748
How does that impact, you know, sort of downstream the actual things that we're talking about here with, um, you know, uh, the Admat and all that stuff?
234
00:42:22.348 --> 00:42:28.988
When we're talking about molecular interaction, the scale of two Angstrom that people typically measure is just too big.
235
00:42:29.628 --> 00:42:37.208
So you can think about, like in image generation model, you would generate a picture, and it's, like, fuzzy. Mm-hmm. So yeah, it's like in general, right?
236
00:42:37.288 --> 00:42:49.800
But you can't discern the details, and the details really matter here. Like with two Angstrom accuracy, the Your entire aromatic ring can be flipped, and it'll still be, uh, a valid, uh, output.
237
00:42:50.080 --> 00:42:59.780
The worst part is that unlike an image which is blurry, uh, you, you know- Right... you don't even know it's blurry, right? [laughs] Yeah. [laughs] You flip, you flip around an aromatic ring- Yeah...
238
00:42:59.780 --> 00:43:06.820
and it looks just fine and, yeah. Or even worse, a heterocyclic aromatic ring, and then you're- Yeah... really in trouble. Yeah. Yeah.
239
00:43:06.860 --> 00:43:17.560
And it really matters in this case because, like, individual atoms need to establish connections here. Um, so that level of accuracy, what they're pushing for, uh, one angstrom is really important.
240
00:43:17.860 --> 00:43:24.180
So maybe it's almost like with a lot of generative image models, they will fill in a detail which just doesn't exist. So- [laughs]...
241
00:43:24.260 --> 00:43:35.540
if you were to try to do, like, forensics and you tell, you know, your, your vision transformer to, to, to enhance an image, and suddenly it just pops up a face, but that's actually not the person's face, right?
242
00:43:36.140 --> 00:43:44.360
But, like, you don't know that, and it feels just fine. And so if you're trying to use this as a structural hypothesis as a med chemist, you're kind of screwed.
243
00:43:44.500 --> 00:43:50.640
If you're looking at the wrong face as an investigator, it's, it's not gonna take... Yeah. Yeah. I really like your framing.
244
00:43:50.820 --> 00:44:00.400
And then as Evan said, at, at, like, at, at as a, as a downstream, you run a bunch of other models- Mm-hmm... like physics-based models, and all of those little issues may compound. Mm-hmm.
245
00:44:00.780 --> 00:44:10.180
Uh, and then obviously, if you made the wrong prediction, like fundamentally wrong, then the downstream predictions are also gonna be wrong. I just wanted to say for your question- Okay...
246
00:44:10.340 --> 00:44:16.760
I really appreciate your mid-2000s TV reference, uh, about- [laughs] Oh... you know, en- enhancing forensic images. Enhanced. Enhanced. So I- Yeah...
247
00:44:16.800 --> 00:44:20.380
just want to say I appreciate that before we get back into, get back in the details here.
248
00:44:21.100 --> 00:44:38.460
We talked about poses, so maybe the, the contrarian take is that a pose isn't even a well-defined concept, in that, you know, the best way of thinking about these things is just like, you know, probability distribution over things, and there's a small molecule which probably lives in a binding pocket, something like this.
249
00:44:38.920 --> 00:44:48.520
I mean, is, is this pose... Like, this is an abstraction that humans use and not really, in some sense, ground truth. So I- I'm actually almost kind of...
250
00:44:48.560 --> 00:44:52.820
I'm kind of interested to actually hear that, like, oh, we have this one angstrom threshold. This is, like, really important for med chemists.
251
00:44:53.420 --> 00:45:01.080
I would also talk to some med chemists who would say, "I mean, this pose isn't even real. This is just, like, maybe a most probable configuration," or you know.
252
00:45:01.740 --> 00:45:06.440
How do you think about that abstraction in general, and do you ex...
253
00:45:06.740 --> 00:45:14.840
Do you explore conformational space and provide that as tools, and, yeah, how, how do you interact with just, like, a single structure versus, like, an ensemble and...
254
00:45:15.620 --> 00:45:23.540
Sorry, I didn't know if that question even makes sense, but... Yes, it's an abstraction, but it's a very useful abstraction. [laughs] It helps us to build up confidence- Mm-hmm...
255
00:45:23.540 --> 00:45:34.680
that a particular model output is actually valid. Mm-hmm. Uh, that you did not just straight up hallucinated something. Mm-hmm. 'Cause yes, ultimately, what matters is binding affinity or potency. Yeah.
256
00:45:34.740 --> 00:45:40.640
And you can straight up predict that with your model- Yeah... and just skip the entire pose generation step.
257
00:45:41.300 --> 00:45:51.400
But when you only have a single number, and that number might as well be completely hallucinated, and you have no means to validate whether that, whether that number even makes any sense. Yeah.
258
00:45:51.460 --> 00:46:02.600
So as much as poses are not perfect- Mm-hmm... they're still a very useful tool- Yeah... uh, for the entire process. I'm sorry. Going in that direction, um, you know, just so you have a frame. No.
259
00:46:02.620 --> 00:46:06.140
I'm jumping into a technical rabbit hole here, but, um- Most yeah... but, like, the...
260
00:46:06.720 --> 00:46:19.360
just because you have a pose also, I mean, there's, like, entropic and enthalpic contributions, and predicting binding affinity is not just about getting the energy right, but actually is this even likely to make it into the binding pocket, and is it likely to live there long term?
261
00:46:19.820 --> 00:46:32.760
So it's much more than just, you know, affinity... Potency prediction is much more than just is this the right pose, but does this have the broader properties it needs to be a long-lived molecule in this state?
262
00:46:33.480 --> 00:46:35.460
And how do you kind of deal with those things too?
263
00:46:36.300 --> 00:46:50.160
A note for sort of the, the wider AI audience, so probably resonate both, both with practitioners as well as users, is, um, you know, obviously something that's blossomed in the, in the past six months is agents.
264
00:46:50.780 --> 00:46:54.700
Um- [laughs] And we love agents. Who doesn't? [laughs] It, it, it's...
265
00:46:54.880 --> 00:47:05.920
There's a lot of, I'm gonna say a few necessary but not sufficient conditions, I'd say, in, in, in my, my sort of re-response to your, to your really, uh, deep question there.
266
00:47:06.380 --> 00:47:19.600
We all remember what agents were like, let's say mid last year. And let's just say there is positive value and there's negative value, and both can be amplified by agents. And why is that?
267
00:47:19.640 --> 00:47:26.440
Well, agents are only as useful as the underlying models that they're orchestrating. And let's think about coding.
268
00:47:26.960 --> 00:47:35.060
If your coding model even makes subtle but, but real bugs, your agents are just going to amplify those issues- Mm-hmm...
269
00:47:35.120 --> 00:47:42.140
and you're gonna end up with not only slop, but something that, that may be anti-useful, that might give the user incorrect information.
270
00:47:42.540 --> 00:47:53.140
We all remember what that, what that age was like mid last year that made a lot of people lose patience, I would say, with claims about LLMs for, for agentic engineering.
271
00:47:53.680 --> 00:48:04.020
Something changed, like clearly a threshold was, was, was, was met, and even though these models are still not perfect, the utility of LLMs for software engineering are so obvious now.
272
00:48:04.300 --> 00:48:17.340
I mean, obviously it's been a huge tailwind and very useful for us, um, for, for replacing a lot of the drudgery o- of coding and getting a focus a little bit more on some of the, the, the more strategic, um, uh, issues that matter.
273
00:48:17.860 --> 00:48:31.840
You could draw a direct analogy from that to what we're talking about here in, in that, um, we'd obviously no... It will be no surprise, but we're working on an agentic platform for twenty-four/seven drug discovery.
274
00:48:31.840 --> 00:48:44.100
If you can imagine just fleets of hundreds of med chemists and cat scientists working nights and weekends all the time for, for your drug targets. Um, the, the, the code name for that gem is Sapphire.
275
00:48:44.820 --> 00:49:06.640
The prerequisite for that was we needed the underlying models for pose, three D, three D complex friction, potency, ADME, to all be good enough for an agent using these models twenty-four seven to create molecules that medicinal chemists would actually want to make and not laugh at.
276
00:49:07.360 --> 00:49:19.320
And if your model is sitting at one point eight, one point nine angstrom RMSD, that's slop, most likely. A-and let me be really direct about it.
277
00:49:20.100 --> 00:49:27.640
What you're hearing, so you're hearing bimodal distribution from people of the utility of a three D pose and how quote real it is.
278
00:49:28.260 --> 00:49:40.180
The reality is that for a highly potent ligand, almost certainly there is a large portion of that molecule with a very well-defined three D position, down to even half angstrom.
279
00:49:40.880 --> 00:49:52.840
And if you don't believe me, you can open up the electron density diagram in the PDB. It's all available online, and you can, you can literally see, in some cases, aromatic rings with missing density in the middle.
280
00:49:53.600 --> 00:50:04.640
So it's not just a construct on a blackboard that your organic chemistry professor showed you. It is-- you can literally see a donut, a torus of electron density for an aromatic ring.
281
00:50:05.180 --> 00:50:16.520
However, there's going to be solvent-exposed areas oftentimes that will be less important for binding affinity, but maybe are important for solubility or, or other properties of your molecule.
282
00:50:17.020 --> 00:50:22.920
Those sometimes are less defined because sometimes in reality, to your point, they're more dynamic. They're flopping around in solvent.
283
00:50:23.660 --> 00:50:41.540
But the critical piece f- as a, as a upstream indicator that's valuable for predicting the free energy of binding is to get the core of your molecule that specifically interacts with the protein to be correct to sub-angstrom resolution.
284
00:50:42.240 --> 00:50:51.660
And why does this matter, just to use it intuitively? So a hydrogen bond. Everyone in the audience has probably heard about a hydrogen bond. It's one of the most critical forms of non-covalent interactions.
285
00:50:52.040 --> 00:50:59.360
It's how nucleic acids are held together. It's how most ligands and proteins interact. It's how proteins form secondary structures.
286
00:50:59.900 --> 00:51:10.220
Hydrogen bonds have a very specific angle and distance, and the distance is from the donor to the acceptor heavy atom. It's two point seven angstroms to three point three angstroms.
287
00:51:10.920 --> 00:51:19.660
And if I do my math right, that's a zero point six angstrom gap. Um, and outside of that, it's not a hydrogen bond. If it's less than that, it's a clash.
288
00:51:19.760 --> 00:51:32.240
If it's more than that, um, the interaction is, is much, much weaker pr-pretty quickly. And angstrom, for those who don't know, is one-tenth of a nanometer. So drug discovery really is a science of resolution.
289
00:51:33.260 --> 00:51:49.020
And if your, um, accuracy is not sufficient, it will therefore not be useful for the downstream things that you care about, which is both potency prediction, but also prospective design. What molecule do I make next?
290
00:51:50.040 --> 00:52:00.900
Um, so that, that's why our view is that the history of innovation in startups is that the ones that do best are ones that focus on one well-defined but very important problem.
291
00:52:01.140 --> 00:52:13.480
And we think our ability to get higher resolution predictions stems from our judicious focus on small and medium-sized molecule design rather than boiling the ocean. Yeah. So that's the what and the why. Yeah.
292
00:52:13.520 --> 00:52:25.420
So what, what about the how? How did you get to one angstrom and sub-one angstrom? I'm gonna give you an extremely boring answer. Okay. [laughs] Uh, which I think is actually true for entire AI field.
293
00:52:25.940 --> 00:52:32.900
Like, where three things mat-mat-matter in AI, um, it's data, infrastructure, and evals. Okay. Right?
294
00:52:33.140 --> 00:52:46.340
So you can only improve what you measure, and once you are very careful about measuring what matters, and you have really talented people on the team, we're gonna figure out how to hill climb that measure, right? Mm-hmm.
295
00:52:46.540 --> 00:52:57.120
So from the start, we actually focused on less one, one, one angstrom precision, and that led to a bunch of small decisions in the process, but compound, right?
296
00:52:57.280 --> 00:53:07.160
If your team never looks into this metric at all, then you will never train a model that, that is good at it. Right. And if you're constantly looking into that, then you're gonna achieve that.
297
00:53:07.280 --> 00:53:13.240
So it's the right objective plus good science and engineering. Correct. Mm-hmm. And it propagates everywhere, right, through the whole stack.
298
00:53:13.600 --> 00:53:22.460
It propagates to how you look at the data, maybe like how do you filter out data. Some data is more noisy, so maybe we don't need to see it, or maybe we don't need to see it later in the training.
299
00:53:23.120 --> 00:53:30.240
It propagates through your modeling architecture and propagates through your loss. How long have you been working towards this specific goal?
300
00:53:30.380 --> 00:53:40.860
I mean, I'm curious, this is not something that you broadly hear in the, the community talk about one angstrom. People do say two-- have decided two angstrom is sort of the- [chuckles] -canonical benchmark.
301
00:53:41.460 --> 00:53:51.480
And I've never heard someone say one angstrom before is the cutoff until, like, reading things coming out of Genesis. How long have you decided this is the number we need to, to hill climb on?
302
00:53:51.980 --> 00:53:58.060
How direct has this focus been? And in, like, the evolution of the company, I'm, I'm just wondering, like, how did this come about?
303
00:53:58.920 --> 00:54:06.640
I think one of our most important secret is that we're working on real drug programs- Mm-hmm... either with partners or in-house.
304
00:54:07.380 --> 00:54:13.800
And when you actually work on real drug programs, you see the failure modes, you see what works and what doesn't work.
305
00:54:14.420 --> 00:54:27.260
And it's pretty obvious when you see at the outputs of real programs what, what kind of failure modes are happening. And it becomes very obvious that two angstrom is just not working out, uh, for those setups. Right.
306
00:54:27.580 --> 00:54:41.220
So, I mean, there's a lot of really smart med chemists who are also, you know, who think very carefully about benchmarks and people I, I respect very much and who also have prosecuted actual drug discovery programs, so I, I'm...
307
00:54:41.600 --> 00:54:44.700
I know, I'm just kinda curious, like, why hasn't this become part of the community?
308
00:54:44.860 --> 00:54:54.880
Is, is this literally just the, the community has never been able to succeed at a one angstrom benchmark, so they kind of settled at two as a, "Okay, it's something we can aspire to," or...
309
00:54:55.400 --> 00:54:59.648
I, I, I know, just- But- Yeah... I mean, to your point, it Sounds like there's a...
310
00:55:00.008 --> 00:55:25.748
In my experience with pharma, there are plenty of problems for which people-- There's kind of these known things amongst the technical experts in a sub-domain, and it-- but that it doesn't get out of pharma, or it doesn't get the attention it deserves because, uh, you know, f-for whatever, some of the information is proprietary and gets kinda passed from company to company but never really released into the public.
311
00:55:26.048 --> 00:55:39.748
Is this your estimation of what's going on? Have you ever heard of SWE-bench? That- Yeah. So, um, Gemini does pretty well on SWE-bench. Sometimes Gemini publishes models that win on some of those software benchmarks.
312
00:55:40.308 --> 00:55:49.508
Raise your hand if you're using Gemini to write code right now instead of, you know, the obvious other name competitors. Yeah, not me for sure. No one. Like, why would you do that?
313
00:55:49.608 --> 00:55:51.868
It's, it's obviously worse in, in practice.
314
00:55:52.548 --> 00:56:04.768
Um, and I, I'd say actually that in this case, if you look at the provenance here of how that happened, RMSD less than two came originally from, a-again, Brent and I have been in the space forever, from docking studies.
315
00:56:04.848 --> 00:56:20.768
L-long before AI models for post-prediction, when physics-based docking studies from academic institutions, 'cause usually large, you know, the so-- the proprietary software makers didn't wanna benchmark their methods against the other methods, so academics had to get licenses and try it.
316
00:56:20.808 --> 00:56:28.868
And, and they introduced RMSD less than two, which is... That's surprising. They're academics. They're not using these things to make drugs. They're, they're using it to, to, to write papers.
317
00:56:29.528 --> 00:56:38.828
And so that got sort of the provenances that, that kind of got repurposed by the AI community. But the actual trend is much more in the direction that we're saying.
318
00:56:38.928 --> 00:56:54.188
So the first, um, uh, big innovation there w-was PoseBusters, we'd say, uh, put out by a, a lab at University of Oxford, which, which pointed out that RMSD itself is insufficient, and we need to look at physical validity as well.
319
00:56:54.728 --> 00:57:01.268
Uh, so that's PoseBusters that I'm talking about, not the benchmark, but the metric. So, so Oxford improved that.
320
00:57:01.308 --> 00:57:11.708
And in the latest release from OpenBind, we just published our benchmarks at Genesis on OpenBind set, which Sergei- Yeah. Well, we'll talk more about that in a bit. Oh, great. Yeah, we fully planned on that set, yeah.
321
00:57:11.728 --> 00:57:22.908
But, but, but if you look in, in their, you know, that's their original publication, their default metrics are using RMSD, PoseBusters validity, and LDDT.
322
00:57:23.288 --> 00:57:32.048
So there's clearly an acknowledgement that's already been made that RMSD less than two is insufficient, and the field is now rapidly evolving to acknowledge that.
323
00:57:32.688 --> 00:57:37.568
You think that the academic literature is going to establish some, some benchmark.
324
00:57:37.608 --> 00:57:49.428
Maybe it's one Angstrom, maybe it's some LDDT or some other metric that will kinda converge and then hopefully, as a community, drive forward these, like, more accurate model, modeling strategies.
325
00:57:49.808 --> 00:57:52.788
I would say there was an eval crisis in our field- Yeah...
326
00:57:52.848 --> 00:58:04.128
that is now in transition right now as our field, which was previously, let's say, a lot more quiet, but if you go to NeurIPS and ICML every year, year after year, the workshops get a lot more crowded.
327
00:58:04.688 --> 00:58:11.488
Those evals are, are in transition now that we realize the, um, flaws of the, of what came before it. Yeah.
328
00:58:11.528 --> 00:58:18.428
That was a really interesting point about, you know, pick the right evals and then just do good machine learning like you normally would do.
329
00:58:19.008 --> 00:58:25.668
But I, I kinda picked up on something Evan said earlier, which is that you started out with, I guess, what you call Potential Net. I don't think we-- you defined that earlier.
330
00:58:25.728 --> 00:58:36.568
But, you know, this graph-based network, and then you did a lot of computational simulations kind of after that. And now you're sort of going back into now once generative modeling took off.
331
00:58:37.188 --> 00:58:48.448
I'm curious about how that evolution worked and how did you find computational techniques to be, like, really crucial to sort of building on the ML for a while?
332
00:58:48.988 --> 00:59:01.508
And did, did that, that sort of computation lead back into generative? Or did you start collecting data to build a generative computational data as, like, this is a way of data documentation for a generative model?
333
00:59:01.908 --> 00:59:06.308
Like, what's sort of the history of that, and then what led to those decisions to the extent you can talk about them?
334
00:59:07.248 --> 00:59:17.628
I will say the, the last line of, of PyTorch I've written is much further back in history than the most recent line of PyTorch that Sergei has committed, so I, I wanna make sure that he gives his, hi-his opinion as well.
335
00:59:18.088 --> 00:59:21.948
Um, I wanna make sure that, um, I, I'm gonna directly answer your question.
336
00:59:22.388 --> 00:59:32.808
I wanna make sure that I address something you said, like, a little bit of time ago that I think kinda got lost, which is you asked about how we do other things, including ADMET prediction. Yeah.
337
00:59:33.368 --> 00:59:42.168
So I just wanna make a comment about that. Just I, I can spend all day, and I'm happy to, diving in details about, about Pearl and the evolution, and, and we will.
338
00:59:42.828 --> 00:59:55.888
Um, I just kinda wanna give it a shout-out to the fact that it, it is one important pillar, but not the only pillar that matters in the trajectory of a drug discovery campaign.
339
00:59:56.648 --> 01:00:07.448
And you mentioned the Potential Net paper, which we're happy to, to see have been, become influential since you and I were working in the space when it was very much in the future, let's say.
340
01:00:07.528 --> 01:00:21.188
But now it, now it is the future, so it's great. Another paper we published around the same time, um, was on, um, neural networks for ADME prediction. And we published two, actually. Sorry, ADME, ADMET.
341
01:00:21.308 --> 01:00:29.268
Can you def- Oh, sorry. Thank you. Yeah. Yeah. Um, absorption, distribution, metabolism, elimination, and toxicity are the five prongs that comprise ADMET.
342
01:00:29.688 --> 01:00:36.228
These are- These are all the properties you talked about before that you just have to get right in order to make a drug successful. Correct. Yeah.
343
01:00:36.268 --> 01:00:57.176
So I would say there's over thirty or so assays Each of which you could imagine if, if you're a neural net person, like a, a multitask neural network or multi-head, it's gotta predict over thirty, you know, three dozen-ish properties, each of which, if it's in the wrong range, means your molecule is not a drug, it's just a tool.
344
01:00:57.876 --> 01:01:16.316
And so these are things like solubility, which Sergey mentioned, which is important for formulation, as in can it be, uh, um, made into a pill that you can take orally, your oral bioavailability, whether or not you're inhibiting certain enzymes called the cytochrome, um, P450s and its different variants.
345
01:01:16.736 --> 01:01:22.056
Um, hERG, a critical channel, which if you inhibit too much, can cause cardiotoxicity.
346
01:01:22.636 --> 01:01:33.996
So these are-- it's an alphabet soup of things that most people here haven't heard of, but- And a lot of these are extremely hard, specifically because it's not like a single, uh, causal effect.
347
01:01:34.056 --> 01:01:44.216
It's, there are oftentimes many processes, many pathways which are involved in defining this one endpoint. So- Correct. And data sets are oftentimes comically small- Right...
348
01:01:44.276 --> 01:01:52.276
um, unless you have, uh, you know, I guess pharma to help you out, but, or at least open source. It's a- It's sparse-... hard, hard problem. It's sparse in the public domain.
349
01:01:52.396 --> 01:02:01.996
I'd say there's actually a range from really directly predictable properties to ones, let's say, that, as you point out, are actually amalgams of other signaling events.
350
01:02:02.036 --> 01:02:09.496
So like whether or not you're hi- you're inhibiting, like CYP3A4, um, that's really specific. It's a certain protein that you're inhibiting.
351
01:02:09.896 --> 01:02:16.616
But- So that's something that Pearl, for example, could actually model and expect to see some, uh, performance here. Yeah. Useful prediction power.
352
01:02:16.676 --> 01:02:27.716
We've been doing 3D work on ADME prediction before anyone else was, and there was a bunch of work that ended up launching, launching the company.
353
01:02:27.896 --> 01:02:39.736
You know, back in the day when AI could still be in peer-reviewed journals, not just sort of like random white papers on, on archive, we published MoleculeNet, which has been cited a few thousands of times at some point.
354
01:02:39.776 --> 01:02:47.756
And we, we also published this paper showing that multitask graph neural networks were the best at the time for doing ADMET prediction on large pharma data sets.
355
01:02:47.796 --> 01:02:52.935
And that, that paper is one of the most cited papers on AI for ADME ever now.
356
01:02:53.196 --> 01:02:59.836
A lot of the papers, as you know, it's like you write them and you kinda move on, but that work, both those works have been quite influential as well. So
357
01:03:00.816 --> 01:03:27.976
our history from the beginning has been to work on not just one problem, but to focus on drug discovery, and in so doing, being able to have the bandwidth to focus on building all of the ML models that are needed for drug discovery and not get distracted by the biological discovery processes that are needed, the target ID side or, or the clinical trial side, but really focus on all those tasks that are required for drug discovery and also molecular generation, right?
358
01:03:28.036 --> 01:03:47.516
These are all important, and I just wanted to make the point at, at the beginning that as you rightly point out, Brandon, Pearl is a 3D structure prediction model, and many of these ADMETs properties can be posed as those, so it is therefore, um, useful in, in not just on target potency prediction as well.
359
01:03:48.096 --> 01:03:55.296
Um, all of them matter, and we've, we've had to, to tar-- to tackle all of them. Do you use Pearl for all these tasks, or they're focused on some of them?
360
01:03:55.296 --> 01:04:06.716
[chuckles] We'll, we'll be sharing some things publicly, um, in the coming period, but I think we've had, we've had enough news of results lately, [chuckles] I think, I think so far. Um, if that makes sense.
361
01:04:06.956 --> 01:04:15.376
Uh, every, every time you publish something, it's actually, it's, it's, it's a lot of work for the team to, to put together. Uh, so we'll, we'll, we'll do that in the coming period.
362
01:04:15.416 --> 01:04:25.756
But the most recent thing we published was obviously the, the OpenBind, um, uh, results, which was a, for a standard 3D, uh, prediction task. So, so one thing I wanna ask about, um, right, kinda keep hitting at this.
363
01:04:25.816 --> 01:04:35.696
So, you know, you talked about there's a lot of synthetic data and then there's, you know, there's ML modeling, which is very kinda generative, modern generative model flavor.
364
01:04:36.376 --> 01:04:48.056
I wanna talk about kind of the feedback between how your computation, your ML, and how wet lab data went into developing this model. And, uh, you know, we talked about priors.
365
01:04:48.656 --> 01:04:59.576
Um, I think maybe one, it's been typically hard to scale, uh, protein ligand, uh, models, uh, well in, in a way which meaningfully generalizes.
366
01:05:00.256 --> 01:05:07.316
And there, you know, this recent Pearl paper and some of your other recent results we'll talk about shortly have shown true generalization.
367
01:05:07.476 --> 01:05:23.176
And what I'm kinda curious about is how do these different aspects feed together to give you this, this power and specifically, um, you know, just saying computation is interesting, but, like, you have to be very careful about computation.
368
01:05:23.376 --> 01:05:34.336
MD has all sorts of biases. If you're not right, it can give you, uh, poor results. Um, and so I'm curious about how all this worked together and especially in the history of the development of the company.
369
01:05:34.396 --> 01:05:36.096
How did you kinda get to this point?
370
01:05:36.556 --> 01:05:55.236
There, there's this, the classic concept of the narrative fallacy where in retrospect, it's, everything seems obvious, and you can draw really linear, you know, processes of how we got from point A to, to point Z, but the reality is always more interesting and much, much messier and much more nonlinear, I think.
371
01:05:56.036 --> 01:06:12.596
And for Sergey and I, who've been training or taming neural nets for, for over a decade, we remember a lot of different eras in the space. And we would've been so excited if we could tell ourselves
372
01:06:13.516 --> 01:06:25.776
the capabilities of our technologies ten years ago, and if we told ourselves how we got there, it might've seemed somewhat obvious, but s- but still, still there are some things that would've been really difficult to foresee.
373
01:06:25.956 --> 01:06:40.176
So one example is, you know, the concept of using generative AI for, for the molecular space is not a new concept, but the reduction of practice would've been very difficult a decade ago.
374
01:06:40.396 --> 01:06:49.812
And so as one concrete example of that- I used to co, uh, co-run the Stanford AI Salon. Uh, it, we had fun organizing.
375
01:06:50.452 --> 01:06:58.312
You know, it's Stanford in the Gates Computer Science building, and we'd get some interesting speakers, and like twenty-five people having wine and cheese, and they're all
376
01:06:59.212 --> 01:07:04.912
n-now, like some very, like, just straight up famous people, right? But back then, AI was a much smaller field.
377
01:07:04.952 --> 01:07:19.092
And I remember very clearly in like twenty seventeen, twenty eighteen talking about GANs and how generative adversarial networks and how, uh, they're clearly the future of image generation, obviously.
378
01:07:19.752 --> 01:07:30.832
And so there was a lot of work then also in, well, can we just apply GANs to produce confirmations of proteins or, or can we try to use them to produce protein ligand poses?
379
01:07:30.872 --> 01:07:41.012
And for all the same reasons that those models were really tricky to train for images, uh, mode collapse was the most famous, um, uh, sort of problem.
380
01:07:41.532 --> 01:07:45.172
They didn't work very well for, for proteins or protein ligand systems.
381
01:07:45.392 --> 01:07:56.172
And we sort of had to wait for the right primitive to get created, and that turned out to be diffusion, which turned out to be a much more useful primitive for the space.
382
01:07:56.892 --> 01:08:05.132
And interesting is actually a lot of image and video models, some, some are using diffusion, but some of those have actually gone autoregressive.
383
01:08:05.892 --> 01:08:22.192
And so what's kind of cool is right now for, for people that are, are interested in really core fundamental AI research, actually some of the most innovative diffusion research is happening in our field, is happening in three-D structure prediction right now.
384
01:08:22.772 --> 01:08:31.392
We-- No one would've predicted that then, but now, like that's, that's kind of a, a pillar of, of diffusion, I'd say. Um, that was unpredictable ten years ago, everything I just said.
385
01:08:31.472 --> 01:08:47.072
And, and in parallel to that, long before the advent of three-D diffusion models for, um, generative tasks in chemistry, um, we were building a variety of tools for the problems of drug discovery at hand.
386
01:08:47.372 --> 01:08:55.632
Some of which were using physics-based methods for predicting potency or, or even for helping predict certain ME properties.
387
01:08:55.832 --> 01:09:08.772
And, uh, same with molecular generation, um, that, that is like, you know, using, um, different techniques to generate new molecular ideas. As Sergey pointed out, there's ten to the sixty drug-like small molecules.
388
01:09:09.332 --> 01:09:23.612
Searching that space efficiently is hard. [laughs] So we've been working on that problem, and those things happened to be available when we wanted to, to take what was then the very nascent
389
01:09:24.572 --> 01:09:38.092
area of co-folding, which was clearly exciting, but not at all ready for prime time, not at all ready for what, um, a, a medicinal chemist would want to use in their day-to-day work, and take it to the realm of
390
01:09:39.152 --> 01:09:45.212
useful and in some cases irreplaceable, like clearly superior to, to a non-co-folding method.
391
01:09:45.792 --> 01:10:03.052
And it kind of just happened that we had been building those other primitives, and so they were available to us, so we could put them together to build out, for example, the, the synthetic data pipeline or, um, uh, to use some of the inference time techniques.
392
01:10:03.092 --> 01:10:11.472
Like we were, we were ready to do those rapidly because we had been working on orthogonal techniques in ways of approaching problems.
393
01:10:11.812 --> 01:10:17.912
Um, so there was some thing here that were-- that was contemplative, but like almost any discovery,
394
01:10:18.912 --> 01:10:38.672
like the discovery of penicillin, I'm not saying what we're doing is on the same order of magnitude or benefit to humanity as, as, as penicillin, but there's always this element of if you're laser focused on a problem for long enough and spend enough time in the lab, or in our case on a computer banging your head a-against the wall on these problems, you can create the luck,
395
01:10:39.592 --> 01:10:46.032
as it were, that will enable some of the developments. How does the lab interact with the development process?
396
01:10:46.932 --> 01:11:00.812
We, as I mentioned before, we train our models on not just structure prediction models, and for those specific models, lab outputs are extremely useful right away, um, like potency prediction, right?
397
01:11:01.012 --> 01:11:09.812
Those outputs can be used directly, uh, for, for model training. Um, but what I'm most excited about going forward is reinforcement learning.
398
01:11:10.392 --> 01:11:17.572
I think that's coming in our field, and we have seen early signs of, uh, it working already with our models.
399
01:11:18.312 --> 01:11:26.952
So where you basically put initially, uh, maybe physics-based feedback to improve your models through RL, uh, typical RL loop.
400
01:11:27.752 --> 01:11:44.392
Uh, but eventually you can go all the way down to the lab, uh, in the loop setups where you're not on-- your model is not only producing like predictions, but you synthesize based on those predictions, and then you measure downstream properties, and then you feed back those predictions into that model.
401
01:11:44.792 --> 01:11:53.932
So how do you get enough volume? I mean, do you have automation to do that, or do you like w- large campaigns that aren't necessarily automated but generate a lot of data?
402
01:11:54.692 --> 01:12:04.712
One thing I'm super proud of is our partnership with Insight. They're such an amazing company. They are extremely, extremely good in, um, producing data.
403
01:12:05.472 --> 01:12:19.132
Basically taking the compounds, generating them, creating them, and then measuring the downstream properties, and then sending those results back. This is such a partnership kind of born in heaven for us- [laughs]...
404
01:12:19.172 --> 01:12:33.872
between Genesis and Insight, where we are able to train models, give predictions, and have results back from Insight super quickly. So y-your like sort of rollouts include a lab, like a, a lab, uh, iteration?
405
01:12:34.272 --> 01:12:45.912
Essentially, yes. Yeah. Okay. That's amazing. Yeah, and diffusion's slow, so it's about the same speed of that sometimes. [laughing] Give it a week. [laughs] It is true. It's like I, I'm a big believer-
406
01:12:47.244 --> 01:13:01.664
That companies are typically really good at one or two things and do best when they can, can focus on it. And in the same way, we've been just really doggedly focused on developing the best AI models for drug discovery.
407
01:13:02.104 --> 01:13:16.304
Like, Insight has that level of maniacal focus on optimizing drug discovery and development. And another sort of trend people like to be talk, talking about is sort of, of course, the rise of, of China in, in biotech.
408
01:13:16.864 --> 01:13:22.164
Um, it's the elephant in the room that- [chuckles]... there's no point avoiding it. It's, it's so in the zeitgeist right now.
409
01:13:22.824 --> 01:13:32.964
And where so, so many Western companies have, you know, had become to rely more on CROs to do a lot of their wet lab work, you know, Insight
410
01:13:33.924 --> 01:13:41.124
became, like, so state-of-the-art in terms of the experimental capabilities they had in-house.
411
01:13:41.644 --> 01:13:52.884
Their productivity is, is, is extremely high, and as Sergey said it, it's a, it's a, it's a match born in heaven for us because what we thrive on is continuous learning of the models.
412
01:13:53.364 --> 01:14:05.664
So we want to have design, make, test, analyze cycles that are as rapid as possible and continuously fine-tune, or in some cases, depending, re- retrain the models based on what we see in the lab.
413
01:14:06.184 --> 01:14:19.284
Um, and so the partnership is, uh, one of, if not the first ever, that enables that sort of true joint foundation model training on historical and, and also prospective data.
414
01:14:19.864 --> 01:14:28.404
And as ML practitioners, we all know here that data is, is just such a critical input in the whole critical ingredient.
415
01:14:28.484 --> 01:14:39.744
So it- it's, it's just extremely exciting for us and, and that's a range, you know, from reflecting our, our conversation so far here, you know, from structure, potency, and a variety of add net properties as well.
416
01:14:40.064 --> 01:14:50.204
Um, um, so, so I think i- immediately will, will improve the strength of our models and, and be really powerful for, for, for generally accelerating drug discovery.
417
01:14:50.824 --> 01:14:56.244
And, and just, uh, uh, we were joking around about the time that it takes to do diffusion.
418
01:14:56.264 --> 01:15:14.804
[chuckles] But, like, w- I don't know, and I don't know if you can disclose this, but how long is it-- does it take to you, you know, you email them with here's some, here's, I don't know how many, ten, hundred compounds, and then they get back to you with the measurements of those realized compounds in what, a day, a week, a month, a hour?
419
01:15:15.164 --> 01:15:23.564
It depends. Like, some compounds are really easy to knock out, really known, like a, a very well-characterized reaction where the, um,
420
01:15:24.764 --> 01:15:35.764
usual conditions just work and, uh, um, sometimes it's, oh, this coupling actually didn't work like the literature said it would, and we have to try different conditions. So, it, it depends.
421
01:15:35.824 --> 01:15:46.824
I'm saying this so the audience understands that for all the claims out there about just like robotic labs, automating synthesis- Yeah. Yeah... the reality is just a lot more complicated than that. Sorry to interrupt.
422
01:15:46.844 --> 01:15:55.224
Yeah. What are some of the complications there? We've talked about- Radically. [chuckles] Yeah, radical. So- Yeah... they all do some sort form of automation.
423
01:15:55.564 --> 01:16:07.604
What, like-- and a- again, not trying to get you to [chuckles] get in some fight with them, but, like, what- [chuckles]... what, what, what's the contrary take here on, okay, what goes wrong with automated lab? Okay.
424
01:16:07.924 --> 01:16:19.504
Yeah. So this is also one of the reasons, as Sergey mentioned, we're, we're really excited about reinforcement learning because it circumvents the problems
425
01:16:20.444 --> 01:16:26.984
of having to do very fast design make, test, analyze cycles from the lab to feed back in the model.
426
01:16:27.084 --> 01:16:37.624
Ideally, we can throw more and more GPUs at the problem and have the model self-train in the same way that we once did for board games in the late twenty tens or, or most recently for coding.
427
01:16:37.684 --> 01:16:40.424
Like, that, that's a key North Star for us.
428
01:16:41.304 --> 01:16:58.964
However, before anyone tells you that there will be a country of geniuses in a data room just solving drug discovery, um, there are some problems in our space that don't lend themselves to RL as, as naturally, and that will require experimental intervention.
429
01:16:59.544 --> 01:17:03.384
And so to dive into the specifics of, of why, so to give some examples and I- Yeah...
430
01:17:03.424 --> 01:17:12.744
so I, you know, coding was always my North Star, what I wanted to do with my life, but I, I spent a fair bit of time myself in the wet lab as a very mediocre experimental scientist.
431
01:17:12.844 --> 01:17:25.884
[laughing] Um, before I found my true joy back in, you know, when, when I got to graduate school, and I can just focus, focus on, on coding all day. Um, the reality is quite messy. So in order to make a new molecule,
432
01:17:26.844 --> 01:17:40.264
um, you have to synthesize it. What that means is different chemical reagents and catalysts and temperatures and solvents coming together in the right way and the right protocol to try to make your new molecule.
433
01:17:40.684 --> 01:17:47.024
And once you do that, you need to purify it. If you don't do that, your positive or negative assay read could be an artifact.
434
01:17:47.424 --> 01:18:01.924
You need to characterize it, so you need to use NMR and other techniques like mass spec to indicate that what you made in your vial is what you actually sought to make in the first place, which is not as trivial as it sounds either.
435
01:18:02.184 --> 01:18:09.684
That's not true for-- only true for us, by the way. For anyone doing material science, for AI- Yes... it's the same sort of challenge where, "Did I make what I actually made?" [chuckles] Yeah.
436
01:18:09.724 --> 01:18:17.824
Is actually a not trivial question at all. Yes. It's easier in small molecule drug discovery than it is in materials, but it's still nontrivial, and it takes, it takes time.
437
01:18:18.404 --> 01:18:31.844
We've seen so many cases where, um, screenings of large molecules sounds great on paper because you can test millions, billions of compounds in one go, so you have more shots on goal.
438
01:18:32.044 --> 01:18:36.184
And, and as a bonus, you have data for your model, right? You just got millions of data points.
439
01:18:36.444 --> 01:18:56.264
While, okay, the reality is that the translation of high throughput screens, whether it's Dell, DNA-encoded libraries or more traditional screens, the R squared of those predictions to, like, the actual business of resynthesizing a molecule de novo and doing a low throughput, high fidelity experiment is a shockingly low R squared.
440
01:18:56.844 --> 01:19:01.864
Um, the false positive rate is enormous for a variety of reasons, um, a- and so-
441
01:19:02.948 --> 01:19:18.468
That's, it's one of the many reasons why people are so excited about AI making those predictions, because in theory, they can be even cleaner than a lot of the wet lab work, especially the high throughput work, and yield to higher enrichment and higher true positive rates.
442
01:19:19.048 --> 01:19:31.188
So- Why can it be more-- Is it just because the, the, the experiments go wrong a lot and even, and even... Does it matter that you have some sort of very precise robot doing it?
443
01:19:31.328 --> 01:19:43.488
It's still, there's too many variables, there's too much slop in the system. It's a few things. So one is that drug discovery frequently involves finding the outliers. Yeah.
444
01:19:44.108 --> 01:19:52.008
One of the many aspects that's so wild about our space is that the properties that one is seeking in a molecule very often anti-correlate.
445
01:19:52.048 --> 01:20:03.728
[inhales] Binding tends to improve the more hydrophobic, the more greasier a compound is, 'cause protein binding pockets are greasy. Guess what? That makes your solubility worse. Yeah.
446
01:20:04.408 --> 01:20:14.468
And then you want to improve your solubility, often by adding polarity to your compound, and suddenly, for those that remember high school biology, cells are surrounded by lipid bilayers.
447
01:20:14.968 --> 01:20:19.148
Your molecule might not get through the cell membrane, 'cause you made your molecule too polar. So
448
01:20:20.148 --> 01:20:27.968
the properties you want often anti-correlate, and that's where you get-- where multi-parameter optimization of molecules ends up feeling like playing Whac-A-Mole.
449
01:20:28.868 --> 01:20:36.948
So you're often searching for real outliers, and that often requires making molecules that are fundamentally novel.
450
01:20:37.548 --> 01:20:51.508
And the kinds of chemistries that today can be automated are fairly constrained, and so your ability to search chemical space broadly for those really top, top Pareto optimal compounds is, is actually very limited.
451
01:20:52.188 --> 01:21:02.188
So the benefits you get in speed have a very harsh trade-off with the actual quality and novelty of the molecules that you can make. So you can do a lot of boring stuff very quickly.
452
01:21:02.248 --> 01:21:05.748
Is that the- [laughs] Uh, if you're lucky, yeah. That would be-- Yeah, if you're lucky.
453
01:21:06.308 --> 01:21:19.348
But to get to the, to actually-- the answer to your, the, the sort of cutting-edge question, do you really-- it's hard to get a robotic system to do those kinds of things. Yes. It-- we would love for that to happen.
454
01:21:19.468 --> 01:21:27.168
It'd be a huge tailwind for what we do, but, um, that's not, that's not- Yeah... available today. Not where we're, we're at today. Yeah. Right.
455
01:21:27.308 --> 01:21:37.288
And so, and so, I, I mean, I know, again, not asking you to disclose anything that you can't, but what is Insight's approach that makes them so fast and effective?
456
01:21:38.148 --> 01:21:51.728
I think if, if one is willing to focus on one problem and say no to distractions- Mm-hmm... it's amazing w-what- Yeah... can happen. Right. And I think one is that their talent density is very, very high,
457
01:21:52.888 --> 01:22:03.888
and they're willing to, to, what is not universally done now in large or medium-sized pharma, which is build so many experimental capabilities in-house. Mm-hmm.
458
01:22:04.468 --> 01:22:15.928
And I actually think that it's possible more companies try to do that given the global dynamic of pharmaceutical development that's changed so dramatically in the, in the last ten years. Right.
459
01:22:16.028 --> 01:22:28.248
So like CRO is taking over everything, then now suddenly it's become commoditized, and then, but you're, also your ca-capability to do the cutting-edge experiments goes down. It can. Um,
460
01:22:29.248 --> 01:22:34.388
it's been a debate for hundreds of years about the benefits of vertical integration. It comes- Yeah...
461
01:22:34.388 --> 01:22:48.128
with all the areas of, of risk and challenges and upfront costs, but has all the benefits of controlling the process end to end. Mm-hmm. When it works, you can do amazing things when you do things end to end. Yeah.
462
01:22:48.228 --> 01:22:57.608
Um, of course, it comes with greater challenges, and I think an appetite, um, uh, needed to, to do it. And this kind of brings me to another question I have, again, about more industry.
463
01:22:57.648 --> 01:23:10.528
But, so you guys-- So maybe now is a good time to talk about your pivot to fr- And, like, so there's name change from Genesis Therapeutics to Genesis Molecular AI, and that that,
464
01:23:11.508 --> 01:23:16.668
I think go, goes hand in hand with maybe a pivot in the strategy of the company.
465
01:23:16.708 --> 01:23:32.308
Or is that in terms of, okay, we're gonna-- Like, what it sounds like to me is we're gonna focus on the AI portion that, and then sell that to pharma rather than be a, you know, sort of a drug company with it also sells the tool.
466
01:23:32.388 --> 01:23:46.388
Is that, is that sort of accurate? When we started the company, um, the origin of the company was fundamental deep learning research methods that we had developed at Stanford. Right.
467
01:23:46.928 --> 01:24:01.148
And the objective we were trying to solve for was: How can we have the greatest impact for patients? And at the time, you know, the, the founding of the company was purely in AI research.
468
01:24:01.788 --> 01:24:19.208
There was no real precedent at that time for, um, a pure model company really, um, making it in the realm of just creating value for partners in pharma or really most legacy industries.
469
01:24:19.808 --> 01:24:32.648
And the other, I think, component was the company being founded on AI research. I think w-we really wanted to show people that we were serious about the biotech domain. Yeah.
470
01:24:32.688 --> 01:24:39.468
So, you know, I think as part of the origin of let's call ourselves Genesis Therapeutics- Yeah... will seem a lot more serious- [laughs]...
471
01:24:39.768 --> 01:24:48.568
to the, to the, to the, uh, you know, all the, the tried and true folks in pharma, the dyed in the wool med chemists, drug developers who we, who we really wanted to work with.
472
01:24:49.008 --> 01:24:54.528
So I think there is an element o-of an attempt at nominative determinism there a bit.
473
01:24:54.548 --> 01:25:06.856
[laughs] Um, but, but also an acknowledgement that I don't think anyone had really shown at that time that you could be a pure AI model company- Yeah... and really make it. In, in the space.
474
01:25:07.236 --> 01:25:19.136
And that was obviously, this is twenty nineteen, so we're-- we had the benefit of being real forebearers, pioneers in the space. We had the downside of being pretty early. Yeah. Thought to be late- [laughs]...
475
01:25:19.156 --> 01:25:20.936
but turns out we're still early innings.
476
01:25:21.656 --> 01:25:48.116
As the company grew and, and evolved, um, we really restructured kind of like a double helix in that we had extremely strong, focused AI research, and we were fortunate to be able to recruit some of the most experienced and accomplished drug hunters, like people on our management team and board who have discovered, co-invented, developed many FDA-approved drugs.
477
01:25:48.656 --> 01:25:56.076
You talk to most med chemists, most chem-most med chemists are-- feel very fortunate if they're able to work on one molecule that gets FDA approval in their whole career.
478
01:25:56.536 --> 01:26:07.136
And we're really fortunate to convince these sort of accomplished drug hunters to come join our management team and, and board at the beginning and, and steer the show. So the outcome has always been the same.
479
01:26:07.276 --> 01:26:12.196
Everything we're talking about, it only matters if we end up helping patients and their families at the end of the day.
480
01:26:12.896 --> 01:26:27.476
And we have felt from the beginning that the, the way to do that at scale is to primarily partner with large, medium-sized pharma companies and biotech companies, so they can do what they're best at, which is novel biology, clinical development.
481
01:26:27.776 --> 01:26:30.876
We can do what we're best at, which is AI, and bring those two together.
482
01:26:31.436 --> 01:26:45.016
In addition to that, as Sergey has said, there's numerous advantages to having in-house wet lab, data generation, clearly, dogfooding the models, getting really direct human feedback from real med chemists what we need to build in an extremely candid way.
483
01:26:45.416 --> 01:26:56.476
And of course, the third thing is that the single most valuable thing in the universe, what I like to joke, is the, the mirror of error said tests. If you ask people, "What is, what is the one thing you most want?"
484
01:26:57.036 --> 01:27:09.696
And you really thought about it, most people would say a medicine for themselves or a family member. And so there is just immense intrinsic value to having one's own pipeline of programs.
485
01:27:10.416 --> 01:27:14.476
And when, when we interact with pharmaceutical partners,
486
01:27:15.596 --> 01:27:29.616
on average, they love that we're working on molecules ourselves because it means that we're not just a bunch of keyboard jockeys, but we live the dream that we are pitching, and we need to use these models in practice for ourselves too.
487
01:27:29.676 --> 01:27:40.456
So they're battle tested in an, in an additional way, and we're sort of domesticated in that we, uh- [laughs] We know what's really difficult about the space. We're, we're not just sort of selling some pipe dream.
488
01:27:41.076 --> 01:27:50.756
It seems to me like there's been a shift in the last, I don't know, year to two years, where suddenly pharma b-- has become interested in buying tools off the shelf.
489
01:27:51.156 --> 01:28:01.636
Whereas previously, there were a lot of companies that started trying to build AI and then had trouble, and so they became pharma companies because that's the only way they could make money, right?
490
01:28:01.716 --> 01:28:11.656
Is you raise a bunch of money and hope that in ten years you have a drug, and if that drug is successful, then you've made it, and otherwise, then you fold or, like, sell off the assets or whatever. And- Yeah...
491
01:28:11.696 --> 01:28:22.676
the-- and it seems like now that's changing. You have a lot of recent, um, AI acquisitions in, you know, in drug discovery, in pathology, in lots of different r-related areas.
492
01:28:23.056 --> 01:28:33.076
And so have you seen this shift, and is that part of the-- Are, are you shifting towards selling models, or is this just a rebranding just because, because of that shift? Like, what's going on?
493
01:28:33.496 --> 01:28:44.276
To finish that, that, that narrative arc there, the, the name change was just to reflect what we were really doing in practice. Okay. Which is we are an AI company as we have been from day negative one- Yeah...
494
01:28:44.376 --> 01:28:58.256
when we were, you know, coding in PyTorch point seventeen, building, you know, the first-- You know, when Sergey was scaling transformers on PyTorch, the first one to do that, and we were scaling graph neural nets in PyTorch, one of the first ones to do that.
495
01:28:58.756 --> 01:29:12.516
Um, so it's reflecting what, what our identity has been from day negative one, um, but also alluding to the fact that we are very serious molecule makers ourselves, um, and, and we are full stack in, in that way.
496
01:29:13.196 --> 01:29:21.796
Um, we are also, to be clear, our mission is to create as many medicines as possible as would otherwise not have been created.
497
01:29:22.236 --> 01:29:31.976
And I think the way to get there is to put our technology directly into the hands of as many drug developers as possible, not just our own, but in big pharma, midcap pharma, and biotech.
498
01:29:32.016 --> 01:29:47.436
And that's partially why we've been building our, our agents so that our models, which are s-- have become so much more powerful, um, can be easily used and orchestrated in the hands of med chemist, cat scientists who might have never written a line of code in their life.
499
01:29:47.836 --> 01:29:54.236
I, I find it interesting to hear that you have now started bringing in agents into your development, um, process.
500
01:29:55.096 --> 01:30:09.116
You know, it sounds like from the very get-go, you-- your philosophy was much more, these are tools for scientists, and you make the best possible tool, and, uh, this lets them optimize whatever pipeline they're gonna use.
501
01:30:09.156 --> 01:30:19.756
But ultimately, this is a tool for medchem. And now you are switching, and you are saying, "We are actually going to have automated systems which make decisions and then advance."
502
01:30:20.476 --> 01:30:31.356
Uh, first of all, you did allude to you've kind of hit maybe the threshold to, you know, the, like, last November's threshold where it just magically became useful. Um, so maybe that's part of it.
503
01:30:31.536 --> 01:30:42.796
But are there other reasons for going in this direction and, like, you know, why, why now? [laughs] Well, let, let me ask you a question in return. Mm-hmm.
504
01:30:42.836 --> 01:30:53.276
How many tools do you think a human med chemist or cat scientist can realistically learn and, like, be proficient in? Oh, I was gonna say, how many can they learn and how many do they use?
505
01:30:53.356 --> 01:31:01.296
Because I've seen that they like to use lots of them, but maybe they're not necessarily good with- Right. But it's really hard, right? Yeah, yeah. A lot of these tools are extremely complex. Mm-hmm.
506
01:31:01.716 --> 01:31:09.236
Uh, they have a lot of parameters that you, that you need to set properly and configure them right. And this is where agents are extremely good at.
507
01:31:09.656 --> 01:31:17.556
Uh, you-- They, they, they actually know how to use all of these tools and how to orchestrate them well. Sorry, wouldn't to interrupt you. When you say agent- Mm-hmm...
508
01:31:17.896 --> 01:31:28.536
then, um, some people like using the term agent to just mean something automated, but other people use agent to specifically mean you have an LLM which is responsible for orchestrating, orchestrating decisions.
509
01:31:29.136 --> 01:31:45.132
Um, I mean In outside of biotech, it's almo-- it's universally the latter, but sometimes in the space of bio, people are still using agents to refer to any sort of decision-bake, based system which is automated out of humans, and maybe some of them are even more like traditional rules based systems or something.
510
01:31:45.532 --> 01:31:52.852
So- Right. So in our, in our case, I mean, uh, AI definition of agents. Okay. Like an LLM, which is able to use all of these tools that we- Mm-hmm...
511
01:31:52.912 --> 01:32:01.012
have been painstakingly building over the many years of existence of this company. Um, and that's a very powerful concept. Mm-hmm.
512
01:32:01.012 --> 01:32:14.872
'Cause now as, as a cat scientist or a med chemist, you don't need to deeply understand all of the details and all of the hyperparameters what you need to set, and you can just let this thing go, uh, and figure out and solve the problems.
513
01:32:15.792 --> 01:32:25.912
Um, and this is where all of the previous questions that we discussed came up, come back as very important one. Like, agents need to be able to understand what these models are producing.
514
01:32:26.372 --> 01:32:27.932
Why do we care about crystal structure?
515
01:32:28.372 --> 01:32:37.912
Well, the agent can actually look at your predicted crystal structure and make decisions based on that, something that we believe is super important, uh, to, to iterate and improve.
516
01:32:38.452 --> 01:32:55.152
You're saying that you can take a crystal structure, uh, which I guess in this case is framed as a graph or like a three-dimensional image [chuckles] that Mol* is interpreting, I mean, in some way, and then it, it actually can make decisions of on what is binding to what and make hypotheses.
517
01:32:55.212 --> 01:33:02.412
Like, you've gotten to that point where you have seen it making effective decisions. Right. Yeah. So there are definitely multiple layers to this- Uh-huh... this question.
518
01:33:02.752 --> 01:33:11.272
You can start with like basic image understanding- Mm-hmm, yeah... right here. Uh, or you can use additional tools to understand what is happening in the crystal structure.
519
01:33:11.912 --> 01:33:22.512
Or you can even train a model which is, which is able to natively understand, um, crystal structure representation and like image modality is a modality for LLMs these days. Mm-hmm.
520
01:33:22.532 --> 01:33:35.292
Crystal structure can be a modality for LLM as well. Okay. You fine-tune it on sequences that look like or that are somehow capturing tokens that are cry- three D crystal structures. Right. Mm-hmm. Yeah.
521
01:33:35.992 --> 01:33:49.212
So if you historically have treated humans, like you were making tools for humans, so now you have agents, how do you reimagine the interaction of humans and agents and the decision-making process to go?
522
01:33:49.732 --> 01:33:57.112
Is this you hope to essentially automate humans out? You want to scale up and do more hypotheses? Or...
523
01:33:57.712 --> 01:34:11.812
Yeah, and then also the kind of the, the, the skeptic would ask, how do you also tell-- make sure that your agents are not making weird decisions where they tweak the wrong hyperparameter, and now you give you the human a bunch of- Yeah...
524
01:34:11.972 --> 01:34:16.332
crap. [laughs] You don't even un- Yeah... you don't even understand the parameter as well enough- Yeah... to know that that happened. Yeah.
525
01:34:16.872 --> 01:34:28.152
It's, it's a great question, and I think the answer here you can see in how coding tools have evolved, right? You remember Cursor when it just appeared, and it was like tap, tap, auto-complete. Mm-hmm.
526
01:34:29.032 --> 01:34:41.812
It was still useful, but not to the extent that it is useful right now, where I can fire up Cursor and it just goes and does everything for me. So I think we're gonna see a very similar trajectory here.
527
01:34:42.592 --> 01:34:54.672
Now, those agents were already useful, but now human need to provide a consistent feedback and basically steer and guide this, this agent along the way. But over time, they're gonna become more independent.
528
01:34:55.312 --> 01:35:05.052
Still, I don't believe in the full automation, like replacing humans. I believe humans are gonna become way more efficient in the job that they're doing using these tools.
529
01:35:05.652 --> 01:35:13.852
So a human will be providing strategic direction to what this agent needs to do, and the agent is gonna be able to go and execute on behalf of a human.
530
01:35:14.672 --> 01:35:29.952
I think this is the best time ever to be a creative human being or really be any human [laughs] because, like, the long, the long arc of history, starting from when we decided that, eh, hunting and gathering, I don't know about that.
531
01:35:30.032 --> 01:35:36.432
I think I'm gonna settle down and, and, and do this whole agriculture thing. I think that started thousands of years of drudgery.
532
01:35:36.432 --> 01:35:44.992
[laughs] Sometimes you wonder why did we actually do agriculture, because everything I've seen seems like agriculture was probably a net loss on the short term. It was...
533
01:35:45.052 --> 01:35:55.692
The only benefit I see in the long run is that if you were a hunter-gatherer and you got sick or born with a genetic illness or got injured, that was like, you're in trouble.
534
01:35:56.052 --> 01:36:00.592
Uh, there's not much more we could do for you. Th-medicine just didn't really exist.
535
01:36:01.372 --> 01:36:13.872
But ultimately, I think the crowning achievement in my mind to civilization is where the sick people among us in society are not seen as burdens, but people who we wanna help.
536
01:36:14.512 --> 01:36:21.912
And that isn't true in the history of the animal kingdom, including humans. Um, it's a sad reality, but it's true.
537
01:36:22.612 --> 01:36:29.392
Whereas now, because of medicine, we, we see people who are sick as people who we want to help, who we can help, and we can help them.
538
01:36:29.892 --> 01:36:34.012
They could become even more productive members of society if that's the thing that you care about.
539
01:36:34.132 --> 01:36:43.672
If to fast-forward that to now, I think that so much of creative work, day-to-day work, if you look at it as a pie chart, time is our most precious resource.
540
01:36:43.732 --> 01:36:55.012
So much of it was being taken up by repetitive tasks that actually didn't require much thought or creativity. And now it's like we've created, like, this additional head space for deep thinking.
541
01:36:55.632 --> 01:37:06.452
And for us, you're solving some of the most technically challenging but, but important problems. We can use these tools to become wildly more productive and creative.
542
01:37:06.972 --> 01:37:17.852
And that goes to the med chemists, drug hunters, cat scientists who are our direct users. They can be grand strategists for a drug discovery campaign.
543
01:37:18.472 --> 01:37:35.172
And when you have hundreds of drug discovery scientists working twenty-four/seven on large data centers, just the output you get in terms of discovery is just gonna be greater, especially if it's used in the hands of expert scientists who know what they're doing.
544
01:37:35.932 --> 01:37:47.252
We probably should have talked about this earlier, but you, uh, have some really exciting results from your, your Pearl model based upon essentially an, a recent open, uh, data set challenge that you alluded to earlier.
545
01:37:47.992 --> 01:38:00.352
Um, sorry we didn't talk about it before, but I would like to give you a chance to talk about this. So there's this open bind. Was it EVA721A protease? Uh, which was a very, very hard target.
546
01:38:00.612 --> 01:38:12.612
Had a lot of things which made it hard for the community or for co-folding methods to generally work and, uh, you ran this internally and I think had some, you know, fun results. Yeah.
547
01:38:12.672 --> 01:38:20.412
I mean, we can also go back to our conversation of, on evals- Mm-hmm... um, and what's wrong with evals in general.
548
01:38:20.952 --> 01:38:29.172
Like, if you look at the open source models and their performance on public benchmarks, it's kind of very similar a-across the board.
549
01:38:29.972 --> 01:38:42.772
Um, and what was surprising when OpenBind came up with way this new benchmark is that you see a wide variety of performance of the models on that. And, well, and part of it is because it's just a single target.
550
01:38:42.872 --> 01:38:53.572
Okay, I get it. But it's also because this is the target that those models haven't seen during the training, so nobody has optimized for that specific target before. Uh, nobody hill climbed on it.
551
01:38:54.372 --> 01:39:05.552
Um, so it was very interesting for us to see how our model is gonna perform out of a box on this particular target that we also have never seen before during training or when we developed these models.
552
01:39:06.392 --> 01:39:16.372
Um, so when it came up, we obviously wanted to look into it. We, we ran Pearl, Pearl system on that, uh, and we were genuinely surprised and excited about results.
553
01:39:17.032 --> 01:39:28.772
Our numbers are way higher than, um, other published, uh, openly available models. And to me, it reflects how Pearl actually behaves in real drug discovery programs.
554
01:39:29.532 --> 01:39:43.751
'Cause we have seen this kind of number differences shifts in internal programs or partnership programs before, but we were unable to talk about them, 'cause a lot of this data is like it's either partnership data or internal data.
555
01:39:43.792 --> 01:39:58.872
We cannot disclose it. Here you have a perfect example where it's an external target, but we didn't know, um, and we just ran on it, um, and we see this staggering difference. Um, so that's kind of what makes me excited.
556
01:39:59.752 --> 01:40:12.672
And there are some specifics of this target, like it has this flexible loop but needs to move, uh, when you insert your ligand in, into the right location, and a lot of methods just are unable to handle it.
557
01:40:13.032 --> 01:40:23.292
And where Pearl was exceptionally good is figuring out how to move this loop. And we are basically correct for every single pose. Uh, like we, we're predicting really well, uh, this loop movement.
558
01:40:24.232 --> 01:40:37.732
Um, yeah, so that's the exciting part. I'm curious about this. What is it about Pearl that allows it to model a dynamic target like that? Pearl model, the way it's trained, right?
559
01:40:38.012 --> 01:40:51.752
It's trained to predict together protein structure and ligand. Um, and like protein structure and ligand together were not necessarily in the same shape or form as the protein in isolation. Yeah.
560
01:40:51.772 --> 01:40:56.112
So the whole training process, it kind of nudges Pearl into that direction.
561
01:40:56.252 --> 01:41:06.092
So are, are there lots of examples in the training set like that where the, there's this dynamic nature where once when, during the binding event, the confirmation changes?
562
01:41:06.492 --> 01:41:15.272
There is induced fit throughout [chuckles] the training set. Yeah, um- This is probably inherited from the, the s- the physics-based simulations that you're using. It certainly helps.
563
01:41:15.812 --> 01:41:25.012
I think the other component that S- that Sergey alluded to was, um, we haven't just been benchmark maxing on the same... We obviously publish in runs and poses.
564
01:41:25.072 --> 01:41:36.772
We published the state-of-the-art numbers in runs and poses last year when we published the Pearl technical report with NVIDIA at GTC at the end of last year. We used runs and poses. We used pose busters.
565
01:41:36.992 --> 01:41:51.192
Pearl does the best on those benchmarks. We did that. Mm-hmm. Um, however, as Sergey is alluding to, the gap from Pearl to next best model is even larger on the, uh, partner drug targets that we work on.
566
01:41:51.772 --> 01:41:58.632
And, um, what that's in part reflection of is the objective of our company is not maxing benchmarks.
567
01:41:58.792 --> 01:42:08.352
The objective of the company is creating value for biotechs and pharma companies that are usually working on hard drug targets that are different from what's in the PDB.
568
01:42:08.852 --> 01:42:21.832
And so we've had to build the models in a way that are able to extrapolate, and the OpenBind challenge was just the latest example of that. Before we wrap up, we always ask two questions.
569
01:42:22.012 --> 01:42:36.552
One is, if you could remove a bottleneck from your industry by fiat- [laughs] Right? So and this is not like for inputs, outputs, uh, business, what, what would that be?
570
01:42:37.292 --> 01:42:46.412
I can say where my main bottleneck right now- Yeah... is, is in GPUs. Okay. Everyone says that [laughs]. Yeah. Look, GPU prices are going up and up and up. Yeah.
571
01:42:46.912 --> 01:42:57.472
And LLM companies are kind of sucking up all of the GPU capacity out there. Um, and I firmly believe that what we are doing is genuinely important for humanity. Mm-hmm.
572
01:42:57.592 --> 01:43:11.372
Like discovering new medicines for, for yourself, for your loved ones is, it is something that we all need at some point in our life. Um, so I think it's very important to unblock that GPU bottleneck for drug discovery.
573
01:43:11.932 --> 01:43:23.272
Is Anthropic actually throttling science because of, uh, [laughs] GPU purchases? So are you getting spicy? [laughs] I don't know if you can answer this, but are you looking at non-NVIDIA...
574
01:43:23.392 --> 01:43:31.312
I mean, I guess you guys are NVIDIA partners or whatever, so but like there... I've seen that some companies are pivoting towards a multi-GPU...
575
01:43:31.352 --> 01:43:39.952
architecture model so that you can take advantage of other suppliers because of the shortage. NVIDIA's been a great supporter of us.
576
01:43:40.012 --> 01:43:54.192
They've invested twice in Genesis, and we collaborate closely, like, you know, not just in terms of the capital, but we optimize kernels together. Um, you know, we co- Pearl was co-authored by Genesis and NVIDIA. They,
577
01:43:55.172 --> 01:44:01.752
um, publicized the Open Mine results with us. We've been very grateful for that. They're a very, very, very big company- Yeah.
578
01:44:01.772 --> 01:44:10.972
[laughs] -and, you know, we're, we're not, uh, the scale of a, of customer like a Meta or something. So we're grateful that they've given us t- time and attention nonetheless.
579
01:44:11.452 --> 01:44:19.712
But I think part of that i- is a self-serving aspect here, which is that obviously, like, n- no one can time, time the market.
580
01:44:20.172 --> 01:44:35.852
But, um, the amount of hype and investment in the traditional LLM space for chatbots, coding sort of thing, at some point, the amount of alpha will not be what's desired compared to the amount of investment.
581
01:44:36.032 --> 01:44:43.172
And I think regardless of the vicissitudes of the economy, medicines will always be high in demand.
582
01:44:43.952 --> 01:45:01.612
And I think whether it's a desire to help humanity or a self-serving interest in looking for the next, the next big thing, I do think that chip makers, including NVIDIA, are going to want to get a lot more invested in life sciences because it will always be high in demand and
583
01:45:02.552 --> 01:45:13.452
the amount of alpha left in pure LLM space is just getting a little questionable. I mean- I mean, all the LLM companies are starting looking into life sciences right now anyway, right? They...
584
01:45:13.972 --> 01:45:26.772
GTP-Rosalind, there's the Claude Initiative. There's lots of startups and whatever. Yep. We think those are all great tailwinds for us. Yeah. And then what, what about you? What, what's the bottleneck besides GPUs?
585
01:45:26.892 --> 01:45:38.172
Oh, I had the same answer as Sergey. [laughs] For, for sure. If I could wave a magic wand, I'd love to have a, just a, a enormous H100, H200 cluster. Uh, that would, that would be, that would be amazing.
586
01:45:38.652 --> 01:45:52.652
And then the second question we ask is just, do you have a call to action for our audience? So meaning, uh, AI engineers plus scientists, um, get involved with something, come apply for jobs. Uh, do...
587
01:45:52.912 --> 01:46:04.352
Like, what's the call to action for them? We are definitely hiring. We are definitely looking for top talent in this space, and I personally, I transitioned from LLM space to drug discovery.
588
01:46:04.452 --> 01:46:11.472
I've never drug- done drug discovery before joining Genesis. I find this field fascinating and extremely interesting.
589
01:46:12.492 --> 01:46:25.552
Um, you know, in LLM companies, you often find research scientists excited about working on architectures, but, like, kind of forced to work on boring stuff. And honestly, LLM architectures are relatively boring.
590
01:46:25.672 --> 01:46:33.812
I, I don't know. I probably alienate half of your audience. [laughs] I think they probably think the same thing. [laughs] But it's like, it, it's a transformer layer invent.
591
01:46:33.872 --> 01:46:41.472
Like, paper was published in 2017, and you go to any LLM lab today, you will see, uh, very, very similar pieces, uh, in their models.
592
01:46:42.192 --> 01:46:50.872
Yes, there is a little bit of flavor of MoEs, but otherwise, architectures are fundamentally very similar to what was discovered back in 2017, right?
593
01:46:51.972 --> 01:46:58.492
Well, our architectures and our models are actually very different and very, very interesting to work with.
594
01:46:59.212 --> 01:47:08.112
So if somebody's excited about working on architectures, well, this space is actually very, very interesting space to work. Yeah. I'd like to comment. Not only are they very...
595
01:47:08.172 --> 01:47:17.352
your architecture is very different from the LLM space, but you're actually... you're also very different from what a lot of the other bio ML space is doing as well.
596
01:47:17.452 --> 01:47:30.832
Like, i- it's, it's, uh, sort of unique both in their subdomain and more broadly. We saw as a recent guest, you had one of the, uh, the, the progenitors of ESM Fold. Yeah. Oh, yeah. Um- Alex Riess. Yeah... uh, yeah.
597
01:47:31.112 --> 01:47:37.012
Th- they cite our model architecture paper that we published and open sourced a few months ago.
598
01:47:37.472 --> 01:47:53.272
So the work that you do here at Genesis, as an AI researcher, there's lots of opportunities for publication, open source, and influencing the field beyond the obvious ways to influence the field by working with top customers that are actually making medicines that change people's lives.
599
01:47:53.432 --> 01:47:59.812
So it's both incredibly intellectually engaging and a lot of opportunities to influence the field and, and get your name out there.
600
01:48:00.352 --> 01:48:10.952
But if you want, you could also just stay being a cog in the machine of an LLM company and- [laughs] No shade, no shade. [laughs] Yeah. [laughs] All right. Well, cool. Thank you so much for- Thank you for chatting...
601
01:48:10.992 --> 01:48:21.612
for coming and speaking with us. I've personally really enjoyed this. Hopefully, the audience did as well. Evan, Sergey, thank you, and we look forward to seeing how things develop.
602
01:48:22.212 --> 01:48:39.332
Um, and hopefully, you get some good job applications out of this. [laughs] Yeah, we, we love the show, so we really appreciate, uh, taking time to, to, to chat. Yeah. Thank you for having us. [outro jingle]