บทถอดความฉบับเต็มของ Ideas: More AI-resilient biosecurity with the Paraphrase Project

thought of there's   00:04:41.440 --> 00:04:45.360 got to be an easier way to do this [LAUGHS]  because it is really difficult to do any type   00:04:45.360 --> 00:04:51.760 of metabolic engineering. And that's how I got  into the AI space, trying to solve these very   00:04:51.760 --> 00:04:58.560 complicated biological problems, trying to build  things that we don't necessarily even understand   00:05:00.160 --> 00:05:04.880 using our understanding from data  or deriving understanding from data. 00:05:04.880 --> 00:05:09.440 So, you know, that's the roundabout way of how I   00:05:09.440 --> 00:05:12.360 got to where I am—the abstract  way of how I got to where I am. 00:05:12.360 --> 00:05:18.560 HORVITZ: And, Tessa, what motivated you  to jump into this area and zoom into   00:05:18.560 --> 00:05:23.360 biology and biosciences and helping  us to avoid catastrophic outcomes? 00:05:23.360 --> 00:05:28.320 ALEXANIAN: Yeah, I mean, probably the origin of me  being really excited about biology is actually a   00:05:28.320 --> 00:05:33.040 book called [The] Lives of the [a] Cell by Lewis  Thomas, which is an extremely beautiful book of   00:05:33.040 --> 00:05:37.600 essays that made me be like, Oh, wow, life is  just incredible. I think I read it when I was,   00:05:37.600 --> 00:05:41.120 you know, 12 or 13, and I was like, Life is  incredible. I want to work on this. This is   00:05:41.120 --> 00:05:47.920 the most beautiful science, right. And then  I, in university, I was studying engineering,   00:05:47.920 --> 00:05:53.200 and I heard there was this engineering team for  engineering biology—this iGEM team—and I joined   00:05:53.200 --> 00:05:58.240 it, and I thought, Oh, this is so cool. I really  got to go work in this field of synthetic biology. 00:05:58.240 --> 00:06:03.280 And then I also tried doing the wet lab biology,  and I was like, Oh, but I don't like this part.   00:06:03.280 --> 00:06:08.640 I don't actually, like, like babysitting microbes.  [LAUGHTER] I think there's a way … some people who   00:06:08.640 --> 00:06:14.400 are great wet lab biologists are made of really  stern stuff. And they really enjoy figuring out   00:06:14.400 --> 00:06:19.120 how to redesign their negative controls so they  can figure out whether it was contamination or   00:06:19.120 --> 00:06:23.200 whether it was, you know, temperature  fluctuation. I'm not that, apparently. 00:06:23.200 --> 00:06:28.400 And so I ended up becoming a lab automation  engineer because I could help the science happen,   00:06:28.400 --> 00:06:31.360 but I … but my responsibilities  were the robots and the computers   00:06:31.360 --> 00:06:34.160 rather than the microbes, which  I find a little bit intransigent. 00:06:34.160 --> 00:06:36.320 HORVITZ: Right. I was thinking of  those tough souls; they also used   00:06:36.320 --> 00:06:40.281 their mouths to do pipetting and so  on of these contaminated fluids … 00:06:40.281 --> 00:06:42.278 WITTMANN: Not anymore.  ALEXANIAN: It's true. [LAUGHTER] 00:06:42.278 --> 00:06:42.309 DIGGANS: Not anymore. [LAUGHS] 00:06:42.309 --> 00:06:45.120 ALEXANIAN: They used to be  tougher. They used to be tougher. 00:06:45.120 --> 00:06:46.440 HORVITZ: James. 00:06:46.440 --> 00:06:50.800 DIGGANS: So I did my undergrad in computer science  and microbiology, mostly because at the time,   00:06:50.800 --> 00:06:54.560 I couldn't pick which of the two I liked more.  I liked them both. And by the time I graduated,   00:06:54.560 --> 00:06:58.720 I was lucky enough that I realized that the  intersection of the two could be a thing. And   00:06:58.720 --> 00:07:03.840 so I did a PhD in computational biology,  and then I worked for five years at the   00:07:03.840 --> 00:07:08.960 MITRE Corporation. It’s a nonprofit. I got the  chance to work with the US biodefense community   00:07:08.960 --> 00:07:15.440 and just found an incredible group of people  working to protect forces and the population   00:07:15.440 --> 00:07:21.680 at large from biological threats and just  learned a ton about both biology and also   00:07:21.680 --> 00:07:26.880 dual-use risk. And then so when Twist called  me and asked if I wanted to join Twist and   00:07:26.880 --> 00:07:31.280 set up their biosecurity program, I leapt at the  chance and have done that for the past 10 years. 00:07:31.280 --> 00:07:32.960 HORVITZ: Well, thanks everyone. 00:07:32.960 --> 00:07:38.720 I believe that AI-powered protein design  in particular is one of the most exciting   00:07:38.720 --> 00:07:45.040 frontiers of modern science. It holds promise  for breakthroughs in medicine, public health,   00:07:45.040 --> 00:07:51.120 even material science. We're already seeing  it lead to new vaccines, novel therapeutics,   00:07:51.120 --> 00:07:56.320 and—on the scientific front—powerful  insights into the machinery of life. 00:07:56.320 --> 00:08:00.240 So there's much more ahead, especially  in how AI can help us promote wellness,   00:08:00.240 --> 00:08:06.720 longevity, and the prevention of disease. But  before we get too far ahead, while some of our   00:08:06.720 --> 00:08:14.560 listeners work in bioscience, many may not have  a good understanding of some of the foundations. 00:08:14.560 --> 00:08:17.760 So, Bruce, can you just give us a  high-level overview of proteins?   00:08:17.760 --> 00:08:22.760 What are they? Why are they important? How do  they figure into human-designed applications? 00:08:22.760 --> 00:08:28.640 WITTMANN: Sure. Yeah. Fortunately, I  used to TA a class on AI for protein   00:08:28.640 --> 00:08:31.393 design, so it’s right in my wheelhouse. [LAUGHS] 00:08:31.393 --> 00:08:31.440 HORVITZ: Perfect, perfect background. [LAUGHS] 00:08:31.440 --> 00:08:35.040 WITTMANN: It's perfect. Yeah. I got  to go back to all of that. Yeah,   00:08:35.040 --> 00:08:41.840 so from the very basic level,  proteins are the workhorses of life. 00:08:41.840 -->

00:08:44.640 Every chemical reaction that  happens in our body—well,   00:08:44.640 --> 00:08:50.000 nearly every chemical reaction that happens in  our body—most of the structure of our cells,   00:08:51.200 --> 00:08:55.360 you name it. Any life process,  proteins are central to it. 00:08:55.360 --> 00:09:00.880 Now proteins are encoded by what are known  as … well, I shouldn't say encoded. They   00:09:00.880 --> 00:09:05.440 are constructed from what are called  amino acids—there are 20 of them—and   00:09:05.440 --> 00:09:09.600 depending on the combination and order in  which you string these amino acids together,   00:09:09.600 --> 00:09:15.520 you get a different protein sequence. So that's  what we mean when we say protein sequence. 00:09:15.520 --> 00:09:22.000 The sequence of a protein then determines  what shape that protein folds into in a cell,   00:09:22.000 --> 00:09:25.920 and that shape determines  what the protein does. So   00:09:25.920 --> 00:09:31.280 we will often say sequence determines  structure, which determines function. 00:09:31.280 --> 00:09:37.520 Now the challenge that we face in engineering  proteins is just how many possibilities there   00:09:37.520 --> 00:09:42.960 are. For all practical purposes, it's  infinite. So we have 20 building blocks.   00:09:43.520 --> 00:09:50.720 There are on average around 300 amino acids  in a protein. So that's 20 to the power of 300   00:09:50.720 --> 00:09:57.040 possible combinations. And a common reference  point is that it's estimated there are around   00:09:57.040 --> 00:10:04.320 10 to the 80 particles in the observable  universe. So beyond astronomical numbers   00:10:04.320 --> 00:10:09.920 of possible combinations that we could have,  and the job of a protein engineer is to find   00:10:09.920 --> 00:10:16.800 that one or a few of the proteins within  that space that do what we want it to do. 00:10:17.760 --> 00:10:25.120 So when a human has an idea of, OK, here's what I  want a protein to do, we have various techniques   00:10:25.840 --> 00:10:31.440 of finding that desired protein, one of  which is using artificial intelligence   00:10:31.440 --> 00:10:37.600 and trying to either sift through  that milieu of potential proteins or,   00:10:37.600 --> 00:10:44.240 as we'll talk about more in this podcast,  physically generating them. So creating   00:10:44.240 --> 00:10:49.800 them in a way, sampling them out of some  distribution of reasonable proteins. 00:10:49.800 --> 00:10:56.160 HORVITZ: Great. So I wanted to throw it to  James now to talk about how protein design   00:10:56.160 --> 00:11:04.400 goes from computer to reality—from in silico to  test tubes. What role does Twist Bioscience play   00:11:04.400 --> 00:11:09.920 in transforming digital protein designs into  synthesized proteins? And maybe we can talk   00:11:09.920 --> 00:11:13.720 also about what safeguards are in place  at your company and why do we need them. 00:11:13.720 --> 00:11:18.720 DIGGANS: So all of these proteins that  Bruce has described are encoded in DNA.   00:11:18.720 --> 00:11:23.200 So the language that our cells use to kind of  store the information about how to make these   00:11:23.200 --> 00:11:29.440 proteins is all encoded in DNA. And so if you  as an engineer have designed a protein and you   00:11:29.440 --> 00:11:33.920 want to test it to see if it does what you  think it does, the first step is to have the   00:11:33.920 --> 00:11:38.960 DNA that encodes that protein manufactured,  and companies like Twist carry out that role. 00:11:39.760 --> 00:11:47.520 So we are cognizant also, however, that these  are what are called dual-use technologies. So   00:11:47.520 --> 00:11:52.560 you can use DNA and proteins for  an incredible variety of amazing   00:11:52.560 --> 00:11:57.508 applications. So drug development,  agricultural improvements,   00:11:57.508 --> 00:12:02.320 bioindustrial manufacturing, all manner of  incredible applications. But you could also   00:12:02.320 --> 00:12:08.640 potentially use those to cause harm so toxins  or other, you know, sort of biological misuse. 00:12:08.640 --> 00:12:13.200 And so the industry has since at least 2010  recognized that they have a responsibility to   00:12:13.200 --> 00:12:18.480 make sure that when we're asked to make some  sequence of DNA that we understand what that   00:12:18.480 --> 00:12:22.880 thing is encoding and who we're giving it  for. So we're screening both the customer   00:12:22.880 --> 00:12:26.160 that's coming to us and we're screening  the sequence that they're requesting. 00:12:26.800 --> 00:12:30.960 And so Twist has long invested in a very,   00:12:31.920 --> 00:12:36.800 sort of, complicated system for essentially  reverse engineering the constructs that we're   00:12:36.800 --> 00:12:40.560 asked to make so that we understand what they  are. And then a system where we engage with   00:12:40.560 --> 00:12:45.040 our customers and make sure that they're going to  use those for legitimate purpose and responsibly. 00:12:45.040 --> 00:12:49.680 HORVITZ: And how do the emergence  of these new generative AI   00:12:49.680 --> 00:12:52.480 tools influence how you think about risk? 00:12:52.480 --> 00:12:56.800 DIGGANS: A lot of the power of these AI  tools is they allow us to make proteins   00:12:56.800 --> 00:13:00.320 or design proteins that have never existed  before in nature to carry out functions   00:13:00.320 --> 00:13:04.240 that don't exist in the natural world.  That's an extremely powerful capability. 00:13:04.240 --> 00:13:10.960 But the existing defensive tools that we use  at DNA synthesis companies generally rely on   00:13:10.960 --> 00:13:15.680 what's called homology, similarity to known  naturally occurring sequences, to determine   00:13:15.680 --> 00:13:20.560 whether something might pose risk. And so AI tools  kind of break the link between those two things. 00:13:20.560

--> 00:13:25.760 HORVITZ: Now you also serve as chair of the  International Gene Synthesis Consortium. Can   00:13:25.760 --> 00:13:31.680 you tell us a little bit more about the IGSC,  its mission, how it supports global biosecurity? 00:13:31.680 --> 00:13:38.720 DIGGANS: Certainly. So the IGSC was founded  in 2010 and right now has grown to more   00:13:38.720 --> 00:13:44.240 than 40 companies and organizations across 10  countries. And the IGSC is essentially a place   00:13:44.240 --> 00:13:50.240 where companies who might be diehard competitors  in the market around nucleic acid synthesis come   00:13:50.240 --> 00:13:56.320 together and design and develop best practices  around biosecurity screening to, kind of, support   00:13:56.320 --> 00:14:00.200 the shared interest we all have in making sure  that these technologies are not subject to misuse. 00:14:00.200 --> 00:14:03.920 HORVITZ: Thanks, James. Now, Tessa,  your organization, IBBIS is focused—it's   00:14:04.800 --> 00:14:08.960 a beautiful mission—on advancing science  while minimizing catastrophic risk,   00:14:08.960 --> 00:14:13.200 likelihood of catastrophic risk.  When we say catastrophic risk,   00:14:13.200 --> 00:14:18.480 what do we really mean, Tessa, in the  context of biology and AI? And how is   00:14:18.480 --> 00:14:23.600 that … do you view that risk landscape as  evolving as AI capabilities are growing? 00:14:23.600 --> 00:14:30.640 ALEXANIAN: I think the … to be honest, as a person  who's been in biosecurity for a while, I've been   00:14:30.640 --> 00:14:35.200 surprised by how much of the conversation about  the risks from advances in artificial intelligence   00:14:35.200 --> 00:14:39.760 has centered on the risk of engineered  biological weapons and engineered pandemics. 00:14:39.760 --> 00:14:46.720 Even recently, there was a new discussion on  introducing redlines for AI that came up at the   00:14:46.720 --> 00:14:51.920 UN General Assembly. And the very first item they  list in their list of risks, if I'm not mistaken,   00:14:51.920 --> 00:14:57.200 was engineered pandemics, which is exactly the  sort of thing that people fear could be done,   00:14:58.640 --> 00:15:02.160 could be done with these biological AI tools. 00:15:02.160 --> 00:15:06.720 Now, I think that when we talk about  catastrophic risk, we talk about,   00:15:06.720 --> 00:15:14.480 you know, something that has an impact on a  large percentage of humanity. And I think the   00:15:14.480 --> 00:15:20.080 reason that we think that biotechnologies pose  a catastrophic risk is that we believe there,   00:15:20.080 --> 00:15:24.480 as we've seen with many historical  pandemics, there's a possibility for   00:15:24.480 --> 00:15:30.800 something to emerge or be created that is  beyond our society's ability to control. 00:15:30.800 --> 00:15:33.120 You know, there were a few countries in COVID that   00:15:33.120 --> 00:15:37.600 managed to more or less successfully do 

a zero-COVID policy, but that was not,   00:15:37.600 --> 00:15:42.240 that was not most countries. That was not  any of the countries that I lived in. And,   00:15:42.240 --> 00:15:45.920 you know, we saw millions of people die. And  I think we believe that with something like   00:15:45.920 --> 00:15:52.400 the 1918 influenza, which had a much higher case  fatality rate, you could have far more people die. 00:15:53.120 --> 00:15:59.200 Now, why we think about this in the context of AI  and where this connects to DNA synthesis is that,   00:15:59.760 --> 00:16:06.560 you know, there is a … these risks of both, sort  of, public health risks, pandemic risks, and   00:16:06.560 --> 00:16:13.040 misuse risks—people deliberately trying to do harm  with biology, as we've seen from the long history   00:16:13.040 --> 00:16:19.760 of biological weapons programs—you know, we think  that those might be accelerated in a few different   00:16:19.760 --> 00:16:25.520 ways by AI technology, both the potential … and  I say potential here because as everyone who has   00:16:25.520 --> 00:16:29.360 worked in a wet lab—which I think is everyone  on this call—knows, engineering biology is really   00:16:29.360 --> 00:16:35.920 difficult. So there's maybe a potential for it to  become easier to develop biological technology for   00:16:35.920 --> 00:16:40.880 the purposes of doing harm, and there's maybe  also the potential to create novel threats. 00:16:40.880 --> 00:16:45.440 And so I think people talk about both of those,  and people have been looking hard for possible   00:16:45.440 --> 00:16:50.560 safeguards. And I think one safeguard that  exists in this biosecurity world that,   00:16:50.560 --> 00:16:53.920 for example, doesn't exist as cleanly in  the cybersecurity world is that none of   00:16:53.920 --> 00:16:59.920 these biological threats can do harm until  they are realized in physical reality,   00:16:59.920 --> 00:17:04.800 until you actually produce the protein or  produce the virus or the microorganism that   00:17:04.800 --> 00:17:11.520 could do harm. And so I think at this point of  production, both in DNA synthesis and elsewhere,   00:17:11.520 --> 00:17:16.320 we have a chance to introduce safeguards that can  have a really large impact on the amount of risk   00:17:16.320 --> 00:17:20.120 that we're facing—as long as we develop those  safeguards in a way that keeps pace with AI. 00:17:20.120 --> 00:17:26.960 HORVITZ: Well, thanks, Tessa. So, Bruce, our  project began when I posed a challenge to you   00:17:26.960 --> 00:17:35.440 of the form: could current open-source AI tools  be tasked with rewriting toxic protein sequences   00:17:35.440 --> 00:17:41.360 in a way that preserves their native structure,  and might they evade today's screening systems? 00:17:41.360 --> 00:17:46.480 And I was preparing for a global workshop on  AI and biosecurity that I'd been organizing   00:17:46.480 --> 00:17:52.240 with Frances Arnold, David Baker, and Lynda  Stuart, and I wanted a concrete case study to   00:17:52.240 --> 00:17:57.360 challenge attendees. And what we found  was interesting and deeply concerning. 00:17:57.360 --> 00:18:03.600 So I wanted to dive in with you, Bruce, on the  technical side. Can you describe some about the   00:18:03.600 --> 00:18:11.040 generative pipeline and how it works and what  you did to build what we might call an AI and   00:18:11.040 --> 00:18:16.920 biosecurity red-teaming pipeline for testing  and securing biosecurity screening tools? 00:18:16.920 --> 00:18:23.440 WITTMANN: Sure. Yeah. I think the best place  to start with this is really by analogy. 00:18:24.240 --> 00:18:31.840 An analogy I often use in this case is the type  of image generation AI tools we're all familiar   00:18:31.840 --> 00:18:39.840 with now where I can tell the AI model, “Hey,  give me a cartoonish picture of a dog playing   00:18:39.840 --> 00:18:46.720 fetch.” And it'll do that, and it'll give us back  something that is likely never been seen before,   00:18:46.720 --> 00:18:54.000 right. That exact image is new, but the  theme is still there. The theme is this dog. 00:18:54.560 --> 00:18:57.600 And that's kind of the same technology that we're   00:18:57.600 --> 00:19:04.000 using in this red-teaming pipeline. Only  rather than using plain language, English,   00:19:04.000 --> 00:19:11.520 we're passing in what we would call conditioning  information that is relevant to a protein. 00:19:11.520 --> 00:19:15.600 So our AI models aren't at the point yet  where I can say, “Give me a protein that   00:19:15.600 --> 00:19:21.920 does x.” That would be the dream. We're a long  way from that. But what instead we do is we pass   00:19:21.920 --> 00:19:29.360 in things that match that theme that we're  interested in. So rather than saying, “Hey,   00:19:29.360 --> 00:19:35.280 give me back the theme on a dog,” we pass  in information that we know will cause or   00:19:35.280 --> 00:19:41.520 at least push this generative model to create a  protein that has the characteristics that we want. 00:19:41.520 --> 00:19:45.840 So in the case of that example you just  mentioned, Eric, it would be the protein   00:19:45.840 --> 00:19:50.320 structure. Like I mentioned earlier,  we usually say structure determines   00:19:50.320 --> 00:19:55.520 function. There's obviously a lot of nuance  to that, but we can, at a first approximation,   00:19:55.520 --> 00:20:00.880 say structure determines function. So if I  ask an AI model, “Hey, here's this structure;   00:20:00.880 --> 00:20:06.640 give me a protein sequence that folds  to this structure,” just like with that   00:20:06.640 --> 00:20:11.440 analogy with the dog, it's going to give  me something that matches that structure   00:20:11.440 --> 00:20:15.280 but that is likely still never been seen  before. It's going to be a new sequence. 00:20:16.560 --> 00:20:21.520 So you can imagine taking

this one step  further. In the red-teaming pipeline, what   00:20:21.520 --> 00:20:27.680 we would do is take a protein that should normally  be captured by DNA synthesis screening—that would   00:20:27.680 --> 00:20:33.520 be captured by DNA synthesis screening—find its  structure, pass it through one of these models,   00:20:33.520 --> 00:20:40.480 and get variants on the theme of that structure  so these new sequences, these synthetic homologs   00:20:40.480 --> 00:20:46.400 that you mentioned, paraphrased, reformulated,  whatever phrase we want to use to describe them. 00:20:46.400 --> 00:20:51.600 And they have a chance or  a greater chance than not   00:20:51.600 --> 00:20:56.240 of maintaining the structure and so  maintaining the function while being   00:20:56.240 --> 00:21:01.120 sufficiently different that they're  not detected by these tools anymore. 00:21:01.120 --> 00:21:06.240 So that's the nuts and bolts of how the  red-teaming pipeline comes together.   00:21:06.240 --> 00:21:11.440 We use more tools than just structure. I think  structure is the easiest one to understand.   00:21:11.440 --> 00:21:19.280 But we have a suite of tools in there, each pass  different conditioning information that causes the   00:21:19.280 --> 00:21:29.360 model to generate sequences that are paraphrased  versions of potential proteins of concern. 00:21:29.360 --> 00:21:34.720 HORVITZ: But to get down to brass tacks, what  Bruce did for the framing study was … we took   00:21:34.720 --> 00:21:42.640 the toxic, well-known toxic protein ricin, as  we described in a framing paper that's actually   00:21:42.640 --> 00:21:51.120 part of the appendix now to the Science  publication, and we generated through   00:21:51.120 --> 00:22:00.000 this pipeline, composed of open-source tools,  thousands of AI-rewritten versions of ricin. 00:22:01.680 --> 00:22:07.200 And this brings us to the next  step of our project, way back when,   00:22:07.200 --> 00:22:12.720 at the early … in the early days of this  effort, where Twist Bioscience was one   00:22:12.720 --> 00:22:20.400 of the companies we approached with what  must have seemed like an unusual question   00:22:21.040 --> 00:22:27.200 to your CEO, in fact, James: would you be  open to testing whether current screening   00:22:27.200 --> 00:22:33.520 systems could detect thousands of AI-rewritten  versions of ricin, a well-known toxic protein? 00:22:33.520 --> 00:22:38.560 And your CEO quickly connected me with you,  James. So, James, what were your first thoughts   00:22:38.560 --> 00:22:45.760 on hearing about this project, and how did  you respond to our initial framing study? 00:22:45.760 --> 00:22:52.000 DIGGANS: I think my first response was gratitude  and excitement. So it was fantastic that Microsoft   00:22:52.000 --> 00:22:58.000 had really leaned forward on this set of ideas  and had produced this dataset. But to have it,   00:22:58.000 --> 00:23:01.200 you know, show up on our doorstep in  a very concrete way with a partner   00:23:01.200 --> 00:23:05.760 that was ready to, sort of, help us try  and address that, I think was a really   00:23:05.760 --> 00:23:11.760 … a valuable opportunity. And  so we really leapt at that. 00:23:11.760 --> 00:23:19.760 HORVITZ: And the results were that both for  you and another company, major producer IDT   00:23:19.760 --> 00:23:26.000 [Integrated DNA Technologies], those thousands  of variants flew through … flew under the radar   00:23:26.000 --> 00:23:31.440 of the biosecurity screening software  as we covered in that framing paper. 00:23:31.440 --> 00:23:38.480 Now, after our initial findings on this,  we quietly shared the paper with a few   00:23:38.480 --> 00:23:43.440 trusted contacts, including some in government.  Through my work with the White House Office of   00:23:43.440 --> 00:23:48.960 Science and Technology Policy, or OSTP, we  connected up with biosecurity leads there,   00:23:48.960 --> 00:23:55.920 and it was an OSTP biosecurity lead who described  our results as the first zero day in AI and   00:23:55.920 --> 00:24:03.200 biosecurity. And now in cybersecurity, a zero day  is a vulnerability unknown to defenders generally,   00:24:03.200 --> 00:24:08.800 meaning there's no time to respond before  it could be exploited should it be known. 00:24:09.440 --> 00:24:14.000 In that vein, we took a cybersecurity  approach. We stood up a CERT—C-E-R-T—a   00:24:14.000 --> 00:24:17.680 cybersecurity [computer] emergency  response team approach used in   00:24:17.680 --> 00:24:24.800 responding to cybersecurity vulnerabilities,  and we implemented this process to address   00:24:24.800 --> 00:24:30.480 what we saw as a vulnerability with  AI-enabled challenges to biosecurity. 00:24:31.840 --> 00:24:36.720 At one point down the line, it was so rewarding  to hear you say, James, “I'm really glad Microsoft   00:24:36.720 --> 00:24:45.920 got here first.” I'm curious how you think about  this kind of AI-enabled vulnerability compared   00:24:45.920 --> 00:24:51.840 to other ones, biosecurity threats, you've  encountered, and I'd love to hear your perspective   00:24:51.840 --> 00:24:56.040 on how we handled the situation from the early  discovery to the coordination and outreach. 00:24:56.040 --> 00:25:01.360 DIGGANS: Yeah, I think in terms  of comparison known threats,   00:25:01.360 --> 00:25:06.640 the challenge here is really there is no good  basis on which we can just, sort of, say, Oh,   00:25:06.640 --> 00:25:12.880 I'll build a new tool to detect this concrete  universe of things, right. This was more a   00:25:12.880 --> 00:25:17.920 pattern of I'm going to use tools—and I love the  name “Paraphrase”; it's a fantastic name—I can   00:25:17.920 --> 00:25:22.880 paraphrase anything that I would normally think  of as biological … as posing biological risk,   00:25:22.880 --> 00:25:29.440 and now that thing is harder to detect for  existing

tools. And so that really was a very   00:25:29.440 --> 00:25:35.760 eye-opening experience, and I think the practice  of forming this CERT response, putting together   00:25:35.760 --> 00:25:41.280 a group of people who were well versed not just  in the threat landscape but also in the defensive   00:25:41.280 --> 00:25:46.960 technologies, and then figuring out how to  mitigate that risk and broaden that study,   00:25:46.960 --> 00:25:52.520 I think, was a really incredibly valuable  response to the entire synthesis industry. 00:25:52.520 --> 00:25:57.760 HORVITZ: Yeah, and, Bruce, can you describe  a little bit about the process by which we   00:25:57.760 --> 00:26:03.360 expanded the effort beyond our initial  framing study to more toxins and then   00:26:03.360 --> 00:26:12.560 to a larger challenge set and then the  results that we pursued and achieved? 00:26:12.560 --> 00:26:17.920 WITTMANN: Yeah, of course. So, you  know, using machine learning lingo,   00:26:17.920 --> 00:26:26.560 you don't want to overfit to a single example. So  early on with this, as part of the framing study,   00:26:26.560 --> 00:26:33.760 we were able to show or I should say James and  coworkers across the screening field were able   00:26:33.760 --> 00:26:39.200 to show that this could be patched, right. We  needed to just make some changes to the tools,   00:26:39.200 --> 00:26:45.840 and we could at the very least detect  ricin or reformulated versions of ricin. 00:26:45.840 --> 00:26:50.880 So the next step of course was then,  OK, how generalizable are these patches?   00:26:50.880 --> 00:26:58.560 Can they detect other reformulated sequences,  as well? So we had to expand the set of   00:26:58.560 --> 00:27:02.960 proteins that we had reformulated. We  couldn't just do 10s of thousands of   00:27:02.960 --> 00:27:11.092 ricins. We had to do 10s of thousands of  name your other potentially hazardous … 00:27:11.092 --> 00:27:12.160 HORVITZ: I think we had 72, was it? 00:27:12.160 --> 00:27:18.800 WITTMANN: It was 72 in the end that we ended up  at. I believe, James, it was you and maybe Jake,   00:27:18.800 --> 00:27:24.143 another one of the authors on the list … on the  paper, who primarily put that list together ... 00:27:24.143 --> 00:27:26.720 HORVITZ: This is Jacob Beal  … Jacob Beal at Raytheon BBN. 00:27:26.720 --> 00:27:35.040 WITTMANN: I think James actually might be the  better one to answer how this list was expanded. 00:27:35.040 --> 00:27:39.600 DIGGANS: Initially the focus [was] on ricin  as a toxin so that list expanded to 62 sort   00:27:39.600 --> 00:27:43.440 of commonly controlled toxins that are  subject to an export control restriction   00:27:43.440 --> 00:27:48.400 or other concern. And then on top of that, we  added 10 viral proteins. So we didn't really   00:27:48.400 --> 00:27:52.880 just want to look at toxins. We also wanted to  look at viral proteins, largely because those   00:27:52.880 --> 00:27:58.560

proteins tend to have multiple functions. They  have highly constrained structures. And so if we   00:27:58.560 --> 00:28:04.240 could work in a toxin context, could Paraphrase  also do the same for viral proteins, as well. 00:28:04.240 --> 00:28:11.280 HORVITZ: And, Bruce, can you describe  some about how we characterize   00:28:11.280 --> 00:28:20.560 the updates and the, we'll say, the boost in  capabilities of the patched screening tools? 00:28:20.560 --> 00:28:27.680 WITTMANN: So we had, like you said, Eric, 72  base proteins or template proteins. And for each   00:28:27.680 --> 00:28:37.840 of those, we had generated a few 100 to a couple  thousand reformulated variants of them. The only   00:28:37.840 --> 00:28:44.640 way to really get any sense of validity of those  sequences was to predict their structures. So we   00:28:44.640 --> 00:28:52.800 predicted protein structures for I think it was  70ish thousand protein structures in the end that   00:28:52.800 --> 00:28:58.080 we had to predict and score them using in silico  metrics. So things like, how similar is this   00:28:58.080 --> 00:29:04.480 to that template, wild-type protein structure  that we used as our conditioning information? 00:29:04.480 --> 00:29:11.520 We put them on a big grid. So we have two axes.  We have on the x-axis—and this is a figure in   00:29:11.520 --> 00:29:16.000 our paper—the quality of the prediction.  It's essentially a confidence metric:   00:29:16.000 --> 00:29:20.000 how realistic is this protein  sequence? And on the other axis   00:29:20.000 --> 00:29:25.680 is, how similar is the predicted structure of  this variant to the original? And ultimately,   00:29:25.680 --> 00:29:30.640 what we were wanting to see was the proteins  that scored well in both of those metrics,   00:29:30.640 --> 00:29:38.480 so that showed up in the top right of that  diagram, were caught primarily, because these are   00:29:38.480 --> 00:29:45.040 again the ones that are most likely, having to say  most likely, to retain function of the original. 00:29:45.040 --> 00:29:52.080 So when you compare the original tools—Tool  Series A, right, the unpatched tools—what   00:29:52.080 --> 00:29:59.360 you'll find is varying degrees of success in the  top right. It varied by tool. But in some cases,   00:29:59.360 --> 00:30:05.760 barely anything being flagged as potentially  hazardous. And so improvement is then in the next   00:30:05.760 --> 00:30:11.800 series—Series B, the patched version of tools—we  have more flagged in that upper-right corner. 00:30:11.800 --> 00:30:19.760 HORVITZ: And we felt confident that we had a  more AI-resilient screening solution across   00:30:19.760 --> 00:30:30.400 the companies, and, James, at this point, the  whole team decided it was time to disclose the   00:30:30.400 --> 00:30:37.520 vulnerability as well as the patch details  and pointers to where to go for the updated   00:30:37.520 --> 00:30:45.440 screening software and to communicate this to  synthesis companies worldwide via the IGSC. This   00:30:45.440 --> 00:30:53.680 was probably July, I think, of 2024. What was  that process like, and how did members respond? 00:30:53.680 --> 00:31:00.480 DIGGANS: I think members were really grateful  and excited. To present to that group, to say,   00:31:00.480 --> 00:31:06.000 hey, this activity (a) has gone on, (b)  was successful, and (c) was kept close   00:31:06.000 --> 00:31:11.440 hold until we knew how to mitigate this, I  think everyone was really gratified by that   00:31:12.000 --> 00:31:15.840 and comforted by the fact that now they had  kind of off-the-shelf solutions that they could   00:31:15.840 --> 00:31:21.760 use to improve their resilience against any  incoming heavily engineered protein designs. 00:31:21.760 --> 00:31:22.880 HORVITZ: Thanks, James. 00:31:22.880 --> 00:31:30.320 Now, I know that we all understand this  particular effort to be important but a   00:31:30.320 --> 00:31:33.920 piece of the biosecurity and  AI problem. I'm just curious   00:31:33.920 --> 00:31:37.280 to … I’ll ask all three of you to  just share some brief reflections. 00:31:37.280 --> 00:31:39.040 I know, Bruce, you've been  on … you’ve stayed on this,   00:31:39.040 --> 00:31:45.280 and we’ve—all of us on the original team—have  other projects going on that are pushing on   00:31:45.280 --> 00:31:50.320 the frontiers ahead of where we were  with this paper when we published it. 00:31:50.320 --> 00:31:55.760 Let me start with Tessa in terms of, like, what  new risks do you see emerging as AI accelerates   00:31:55.760 --> 00:32:01.920 and maybe couple that with thoughts about  how do we proactively get ahead of them. 00:32:01.920 --> 00:32:09.920 ALEXANIAN: Yeah, I think with the Paraphrase’s  work, as Bruce explained so well, you know,   00:32:09.920 --> 00:32:14.480 I sometimes use the metaphor of the  previous response that the IGSC had to do,   00:32:14.480 --> 00:32:19.200 the synthesis screening community, where it  used to be you could look for similarities to   00:32:19.200 --> 00:32:24.640 DNA sequences, and then everyone started doing  synthetic biology where they were doing codon   00:32:24.640 --> 00:32:29.840 optimization so that proteins could express  more efficiently in different host organisms,   00:32:29.840 --> 00:32:34.320 and now all of a sudden, well, you've  scrambled your DNA sequence and it doesn't   00:32:34.320 --> 00:32:37.840 look very similar even though your protein  sequence actually still looks, you know,   00:32:37.840 --> 00:32:43.120 very similar or often the same once it's been  translated from DNA to protein, and so that was a,   00:32:44.400 --> 00:32:48.560 you know, many, many in the industry were  already screening both DNA and protein,   00:32:48.560 --> 00:32:52.240 but they had to start screening … everybody  had to start screening protein sequences   00:32:52.240 --> 00:32:57.680 even just to do

the similarity testing as these  codon optimization tools became universal. 00:32:57.680 --> 00:33:02.880 I feel like we're, kind of, in a similar  transition phase with protein-design,   00:33:02.880 --> 00:33:09.600 protein-rephrasing, tools where, you know, these  tools are still in many cases drawing from the   00:33:09.600 --> 00:33:16.080 natural distribution of proteins. You know,  I think some of the work we saw in, you know,   00:33:16.080 --> 00:33:20.960 designing novel CRISPR enzymes, you go, OK,  yeah, it is novel; it's very unlike any one   00:33:20.960 --> 00:33:26.320 CRISPR enzyme. But if you do a massive multiple  sequence alignment of every CRISPR enzyme that   00:33:26.320 --> 00:33:31.120 we know about, you're like, OK, this fits in  the distribution of those enzymes. And so,   00:33:31.120 --> 00:33:36.960 you know, I think we're not … we're having  to do a more flexible form of screening,   00:33:36.960 --> 00:33:40.720 where we look for things that are kind of  within distribution of natural proteins. 00:33:40.720 --> 00:33:45.040 But I feel like broadly, all of the screening  tools were able to respond by doing something   00:33:45.040 --> 00:33:52.160 like that. And I think … I still feel like the  clock is ticking down on that and that as the   00:33:52.800 --> 00:33:56.000 AI tools get better at predicting  function and designing, sort of,   00:33:57.520 --> 00:34:00.800 novel sequences to pursue a particular  function, you know—you have tools now   00:34:00.800 --> 00:34:07.040 that can go from Gene Ontology terms to a  potential structure or potential sequence   00:34:07.040 --> 00:34:12.320 that may again be much farther out of the  distribution of natural protein—I think   00:34:12.320 --> 00:34:15.280 all of us on the screening side are going  to have to be responding to that, as well. 00:34:15.280 --> 00:34:22.000 So I think I see this as a necessary ongoing  engagement between people at the frontier of   00:34:24.880 --> 00:34:31.280 designing novel biology and people at the frontier  of producing all of the materials that allow that   00:34:31.280 --> 00:34:40.000 novel biology to be tested in the lab. You know,  I think this feels like the first, you know,   00:34:40.000 --> 00:34:44.960 detailed, comprehensive zero day disclosure and  response. But I think that's … I think we're   00:34:44.960 --> 00:34:50.560 going to see more of those. And I think what  I'm excited about doing at IBBIS is trying to   00:34:50.560 --> 00:34:54.960 encourage and set up more infrastructure  so that you can, as an AI developer,   00:34:54.960 --> 00:35:00.280 disclose these new discoveries to the people who  need to respond before the publication comes out. 00:35:00.280 --> 00:35:02.000 HORVITZ: Thank you, Tessa. 00:35:02.000 --> 00:35:06.160 The, the … Bruce, I mean, you and I  are working on all sorts of dimensions.   00:35:06.160 --> 00:35:12.960 You're leading up some efforts at Microsoft, for  example, on the foundation model front and so on,   00:35:12.960 --> 00:35:17.360 among other directions. We've talked about new  kinds of embedding models that might go beyond   00:35:17.360 --> 00:35:25.920 sequence and structure. Can you talk a little bit  about just a few of the directions that just paint   00:35:25.920 --> 00:35:33.280 the larger constellation of the kinds of things  that we talk about when we put our worry hats on? 00:35:33.280 --> 00:35:35.760 WITTMANN: I feel like that could  have its own dedicated podcast,   00:35:35.760 --> 00:35:39.174 as well. There's a lot … [LAUGHTER]  there's a lot to talk about. 00:35:39.174 --> 00:35:40.320 HORVITZ: Yeah. We want to make sure that we   00:35:40.320 --> 00:35:44.240 don't tell the world that the  whole problem is solved here. 00:35:44.240 --> 00:35:48.560 WITTMANN: Right, right, right.  I think Tessa said it really,   00:35:48.560 --> 00:35:56.000 really well in that most of what we're doing  right now, it's a variant on a known theme.   00:35:57.680 --> 00:36:03.200 I have to know the structure that does something  bad to be able to pass it in as context. I have   00:36:03.200 --> 00:36:07.360 to know some existing sequence that  does something bad to pass it in. 00:36:07.360 --> 00:36:15.200 And obviously the goal is to move away from that  in benign applications, where when I'm designing   00:36:15.200 --> 00:36:20.960 something, I often want to design it because  nothing exists [LAUGHS] that already does it.   00:36:20.960 --> 00:36:30.480 So we are going to be heading to this space  where we don't know what this protein does.   00:36:31.040 --> 00:36:35.360 It's kind of a circular problem, right,  where we're going to need to be able to   00:36:35.360 --> 00:36:45.920 predict what some obscure protein sequence does  in order to be able to still do our screening. 00:36:45.920 --> 00:36:51.440 Now, the way that I think about this,  I often think about it beyond just DNA   00:36:51.440 --> 00:36:58.080 synthesis screening. It's one line of defense,  and there needs to be many lines of defense   00:36:58.080 --> 00:37:06.400 that come into play here that go beyond  just relying on this one roadblock. It's   00:37:06.400 --> 00:37:12.160 a very powerful roadblock. It's a very powerful  barrier. But we need to be proactively thinking   00:37:12.160 --> 00:37:18.720 about how we broaden the scope of defenses.  And there are lots of conversations that   00:37:18.720 --> 00:37:23.920 are ongoing. I won't go into the details of  them. Again, that would be its own podcast. 00:37:23.920 --> 00:37:30.320 But primarily my big push—and I think  this is emerging consensus in the field,   00:37:30.320 --> 00:37:37.920 though I don't want to speak for everybody—is it  needs to … any interventions we have need to come   00:37:37.920 --> 00:37:46.240 more at the systems level and less at the model  level, primarily because this is such dual-use   00:37:46.240 -->

00:37:53.120 technology. If it can be used for good biological  design, it can be used for bad biological design.   00:37:53.120 --> 00:37:59.440 Biology has no sense of morality. There  is no bad protein. It's just a protein. 00:37:59.440 --> 00:38:04.960 So we need to think about this differently than  how we would maybe think about looking at the   00:38:04.960 --> 00:38:10.320 outputs of that image generator model that I  spoke about earlier, where I can physically   00:38:10.320 --> 00:38:14.720 look at an image and say, don't want my model  producing that, do want my model producing   00:38:14.720 --> 00:38:19.920 that. I don't have that luxury in this space.  So it's a totally different problem. It's an   00:38:19.920 --> 00:38:25.680 evolving problem. Conversations are happening  about it, but the work is very much not done. 00:38:25.680 --> 00:38:30.560 HORVITZ: And, James, I want to  give you the same open question,   00:38:30.560 --> 00:38:34.480 but I'd like to apply what Bruce  just said on system level and so on   00:38:34.480 --> 00:38:38.640 and in the spirit of the kind of things that  you're very much involved with internationally   00:38:39.520 --> 00:38:46.080 to also add to it, just get some comments on  programs and policies that move beyond technical   00:38:46.080 --> 00:38:54.640 solutions for governance mechanisms—logging,  auditing nucleic acid orders, transparency,   00:38:54.640 --> 00:39:00.000 various kinds—that might complement technical  approaches like Paraphrase and their status today. 00:39:00.000 --> 00:39:03.760 DIGGANS: Yeah, I'm very gratified that  Bruce said that we, the synthesis industry,   00:39:03.760 --> 00:39:08.880 should not be the sole bulwark against  misuse. That is very comforting and correct. 00:39:08.880 --> 00:39:14.000 Yeah, so the US government published a guidance  document in 2023 that essentially said you,   00:39:14.000 --> 00:39:17.200 the entire biotech supply chain, have  a responsibility to make sure that   00:39:17.200 --> 00:39:19.920 you're evaluating your customers.  You should know your customer;   00:39:19.920 --> 00:39:22.800 you know that they're legitimate. I  think that's an important practice. 00:39:23.680 --> 00:39:29.120 Export controls are designed to minimize  the movement of equipment and materials   00:39:29.120 --> 00:39:34.800 that can be used in support of these  kinds of misuse activities. And then   00:39:34.800 --> 00:39:38.880 governments have really been quite  active in trying to incentivize,   00:39:40.000 --> 00:39:43.760 you know, sort of what we would think  of as positive behavior, so screening,   00:39:43.760 --> 00:39:49.520 for example, in DNA synthesis companies. The  US government created a framework in 2024,   00:39:49.520 --> 00:39:54.880 and it's under a rewrite now to basically say US  research dollars will only go to companies who   00:39:54.880 --> 00:40:01.920 make nucleic acid who do these good things. And  so that is using, kind of, the government-funding   00:40:01.920 --> 00:40:08.560 carrot to, kind of, continue to build these  layers of defense against potential misuse. 00:40:08.560 --> 00:40:13.600 HORVITZ: Thanks. Now, discussing risk,  especially when it involves AI and biosecurity,   00:40:13.600 --> 00:40:20.160 isn't always easy. As we've all been suggesting,  some worry about alarming the public or arming   00:40:20.160 --> 00:40:26.720 bad actors. Others advocate for openness as  a principle of doing science with integrity. 00:40:27.360 --> 00:40:32.800 A phase of our work as we prepared our paper was  giving serious thought to both the benefits and   00:40:32.800 --> 00:40:38.160 the risks of transparency about what it was  that we were doing. Some experts encouraged   00:40:38.160 --> 00:40:43.440 full disclosure as important for enhancing  the science of biosecurity. Other experts,   00:40:43.440 --> 00:40:50.880 all experts, cautioned against what are  called information hazards, the risk of   00:40:50.880 --> 00:40:57.680 sharing the details to enable malevolent  actions with our findings or our approach. 00:40:57.680 --> 00:41:02.720 So we faced a real question: how can we  support open science while minimizing   00:41:02.720 --> 00:41:07.760 the risk of misuse? And we took all the  input we got, even if it was contradictory,   00:41:07.760 --> 00:41:14.320 very seriously. We carefully deliberated  about a good balance, and even then, once   00:41:14.320 --> 00:41:20.640 we chose our balance and submitted our manuscript  to Science, the peer reviewers came back and said   00:41:20.640 --> 00:41:27.360 they wanted some of the more sensitive details  that we withheld with explanations as to why. 00:41:27.360 --> 00:41:30.240 So this provoked some thinking out  of the box about a novel approach,   00:41:30.240 --> 00:41:37.360 and we came up with a perpetual gatekeeping  strategy where requests for access to sensitive   00:41:37.360 --> 00:41:42.400 methods and data and even the software across  different risk categories would be carefully   00:41:42.400 --> 00:41:51.040 reviewed by a committee and a process for  access that would continue in perpetuity. 00:41:51.040 --> 00:41:56.560 Now, we brought the proposal to Tessa and her  team at IBBIS—this is a great nonprofit group;   00:41:56.560 --> 00:42:01.280 look at their mission—and we worked  with Tessa and her colleagues   00:42:03.840 --> 00:42:08.560 to refine a workable solution that  was accepted by Science magazine   00:42:08.560 --> 00:42:14.720 as a new approach to handling information  hazards as first demonstrated by our paper. 00:42:14.720 --> 00:42:19.200 So, Tessa, thank you again for helping  us to navigate such a complex challenge.   00:42:19.200 --> 00:42:22.320 Can you share your perspective on  information hazards? And then walk us   00:42:22.320 --> 00:42:27.560 through how our proposed system ensures  responsible data and software sharing.

00:42:27.560 --> 00:42:29.840 ALEXANIAN: Yeah. And thanks, Eric. 00:42:29.840 --> 00:42:34.640 It's all of the long discussions we had among  the group of people on this podcast and the   00:42:34.640 --> 00:42:39.360 other authors on the paper and many people  we engaged, you know, technical experts,   00:42:39.360 --> 00:42:44.720 people in various governments, you know,  we heard a lot of contradictory advice. 00:42:44.720 --> 00:42:52.320 And I think it showed us that there isn't a  consensus right now on how to handle information   00:42:52.320 --> 00:42:57.840 hazards in biotechnology. You know, I think … I  don't want to overstate how much of a consensus   00:42:57.840 --> 00:43:01.760 there is in cybersecurity either. If you go to  DEF CON, you'll hear people about how they've been   00:43:01.760 --> 00:43:06.640 mistreated in their attempts to do responsible  disclosure for pacemakers and whatnot. But I   00:43:06.640 --> 00:43:11.200 think we're … we have even less of a consensus  when it comes to handling biological information. 00:43:11.200 --> 00:43:16.480 You know, you have some people who say, oh,  because the size of the consequences could be   00:43:16.480 --> 00:43:24.880 so catastrophic if someone, you know, releases  an engineered flu or something, you know,   00:43:24.880 --> 00:43:28.480 we should just never share information about this.  And then you have other people who say there's no   00:43:28.480 --> 00:43:33.680 possibility of building defenses unless we share  information about this. And we heard very strong   00:43:33.680 --> 00:43:38.400 voices with both of those perspectives  in the process of conducting this study. 00:43:38.400 --> 00:43:43.760 And I think what we landed on that I'm really  excited about and really excited to get feedback   00:43:43.760 --> 00:43:49.440 on now that the paper is out, you know, if you  go and compare our preprint, which came out in   00:43:49.440 --> 00:43:55.520 December of 2024, and this paper in October 2025,  you'll see a lot of information got added back in. 00:43:55.520 --> 00:44:00.960 And I'm excited to see people's reaction  to that because even back in January 2025,   00:44:00.960 --> 00:44:06.960 talking with people who were signatories to the  responsible biodesign commitments, they were   00:44:06.960 --> 00:44:13.200 really excited that this was such an empirically  concrete paper because they'd maybe read a number   00:44:13.200 --> 00:44:18.640 of papers talking about biosecurity risks from  AI that didn't include a whole lot of data,   00:44:19.440 --> 00:44:22.960 you know, often, I think, because of concerns  about information hazards. And they found   00:44:22.960 --> 00:44:27.360 the arguments in this paper are much more  convincing because we are able to share data. 00:44:27.360 --> 00:44:33.200 So the process we underwent that I felt good  about was trying to really clearly articulate   00:44:33.200 --> 00:44:36.960 when we talk about

an information hazard,  what are we worried about being done with   00:44:36.960 --> 00:44:41.440 this data? And if we put this data  in public, completely open source,   00:44:41.440 --> 00:44:46.960 does it shift the risk at all? You know, I  think doing that kind of marginal contribution   00:44:48.560 --> 00:44:53.120 comparison is really important because it also  let us make more things available publicly. 00:44:53.120 --> 00:44:57.440 But there were a few tiers of data that after  a lot of discussion amongst the authors of the   00:44:57.440 --> 00:45:03.200 paper, we thought, OK, potentially someone who  wanted to do harm, if they got access to this   00:45:03.200 --> 00:45:07.920 data, it might make it easier for them. Again,  not necessarily saying it, you know, it opens   00:45:07.920 --> 00:45:12.080 the floodgates, but it might make it easier for  them. And when we thought about that, we thought,   00:45:12.080 --> 00:45:18.240 OK, you know, giving all of those paraphrased  protein sequences, maybe, maybe that, you know,   00:45:18.240 --> 00:45:21.600 compared to having to set up the whole pipeline  with the open-source tools yourself, just giving   00:45:21.600 --> 00:45:25.920 you those protein sequences, maybe that makes your  life a bit easier if you're trying to do harm. 00:45:25.920 --> 00:45:29.520 And then we thought, OK, giving you those  protein sequences plus whether or not they   00:45:29.520 --> 00:45:33.920 were successfully flagged, maybe that makes  your life, you know, quite a bit easier.   00:45:33.920 --> 00:45:37.520 And then finally, we thought, OK, the code that  we want to share with some people who might try   00:45:37.520 --> 00:45:41.920 to reproduce these results or might try to build  new screening systems that are more robust,   00:45:41.920 --> 00:45:45.680 we want to share the code with them. But  again, if you have that whole code pipeline   00:45:45.680 --> 00:45:50.800 just prepared for you, it might really help make  your life easier if you're trying to do harm. 00:45:50.800 --> 00:45:56.160 And so we, sort of, sorted the data into these  three tiers and then went through a process   00:45:56.160 --> 00:46:01.920 actually very inspired by the existing customer  screening processes in nucleic acid synthesis   00:46:01.920 --> 00:46:08.160 about how to determine, you know, we tried to  take an approach not of what gets you in but   00:46:08.160 --> 00:46:13.440 what gets you out. You know, for the most part, we  think it should be possible to access this data. 00:46:13.440 --> 00:46:18.640 You know, if you have an affiliation with a  recognizable institution or some good explanation   00:46:18.640 --> 00:46:23.920 of why you don't have one right now, you know,  if you have a reason for accessing this data,   00:46:23.920 --> 00:46:28.320 it shouldn't be too hard to meet those  requirements, but we wanted to have some   00:46:28.320 --> 00:46:32.240 in place. And we wanted it to be possible to  rule out some people from getting access to   00:46:32.240 --> 00:46:36.400 this data. And so we've tried to be extremely  transparent about what those are. If you go   00:46:36.400 --> 00:46:40.800 through our data access process and for some  reason you get rejected, you'll get a list of,   00:46:40.800 --> 00:46:44.400 “Here's the reasons we rejected you. If you  don't think that's right, get back to us.” 00:46:45.040 --> 00:46:50.400 So I'm really excited to pilot this in part  because I think, you know, we're already in   00:46:50.400 --> 00:46:55.840 conversations with some other people handling  potential bio-AI information hazard about doing   00:46:55.840 --> 00:47:00.560 a similar process for their data of, you know,  tiering it, determining which gates to put in   00:47:00.560 --> 00:47:05.840 which tiers, but I really hope a number of people  do get access through the process or if they try   00:47:05.840 --> 00:47:13.040 and they fail, they tell us why. Because I think  as we move toward this world of potentially,   00:47:13.040 --> 00:47:19.200 you know, biology that is much easier to engineer,  partly due to dual-use tools, you know, my dream   00:47:19.200 --> 00:47:24.320 is it's, like, still hard to engineer harm with  biology, even if it's really easy to engineer   00:47:24.320 --> 00:47:31.440 biology. And I think these, kind of, new processes  for managing access to things, this sort of like,   00:47:31.440 --> 00:47:37.480 you know, open but not completely public, I think  those can be a big part of that layered defense. 00:47:37.480 --> 00:47:41.840 HORVITZ: Thanks, Tessa. So we're getting close  to closing, and I just thought I would ask each   00:47:41.840 --> 00:47:51.600 of you to just share some reflections on what  we've learned, the process we've demonstrated,   00:47:52.880 --> 00:47:58.960 the tools, the policy work that we did,  this idea of facing the dual-use dilemma   00:47:59.520 --> 00:48:05.360 with … even at the information hazard level,  with sharing information versus withholding it.   00:48:07.040 --> 00:48:11.680 What do you think about how our  whole end to end of the study,   00:48:11.680 --> 00:48:17.920 now reaching the two-year point, can help  other fields facing dual-use dilemmas? 00:48:17.920 --> 00:48:24.160 Tessa, Bruce, James … James, have you ever thought  about that? And we'll go to Bruce and then Tessa. 00:48:24.160 --> 00:48:30.160 DIGGANS: Yeah, I think it was an excellent  model. I would like to see a study like this   00:48:30.160 --> 00:48:34.960 repeated on a schedule, you know, every six  months because from where I sit, you know,   00:48:34.960 --> 00:48:40.480 the tools that we used for this project are now  two years old. And so capabilities have moved on.   00:48:40.480 --> 00:48:46.320 Is the picture the same in terms of defensive  capability? And so using that model over time,   00:48:46.320 --> 00:48:51.280 I think, would be

incredibly valuable.  And then using the findings to chart,   00:48:51.280 --> 00:48:56.160 you know, how much should we be investing  in alternative strategies for this kind of   00:48:56.160 --> 00:48:59.200 risk mitigation for AI tool  … the products of AI tools? 00:48:59.200 --> 00:48:59.840 HORVITZ: Bruce. 00:48:59.840 --> 00:49:06.480 WITTMANN: Yeah, I think I would extend on  what James said. The anecdote I like to   00:49:06.480 --> 00:49:11.040 point out about this project is, kind of,  our schedule. We found the vulnerability   00:49:11.040 --> 00:49:17.840 and it was patched within a week, two weeks,  on all major synthesis screening platforms.   00:49:18.880 --> 00:49:25.120 We wrote the paper within a month. We expanded  on the paper within two months, and then we spent   00:49:25.120 --> 00:49:29.280 a year and a half to nearly two years [LAUGHS]  trying to figure out what goes into the paper;   00:49:29.280 --> 00:49:34.480 how do we release this information; you  know, how do we do this responsibly? 00:49:34.480 --> 00:49:44.080 And my hope is similar to what James said.  We've made it easier for others to do this   00:49:44.080 --> 00:49:48.480 type of work. Not this exact work; it doesn't  have to necessarily do with proteins. But to do   00:49:48.480 --> 00:49:55.360 this type of work where you are dealing with  potential hazards but there is also value in   00:49:55.360 --> 00:50:02.960 sharing and that hopefully that year and a half  we spent figuring out how to appropriately share   00:50:02.960 --> 00:50:08.640 and what to share will not be a year and a  half for other teams because these systems   00:50:08.640 --> 00:50:14.440 are in place or at least there is an example  to follow up from. So that's my takeaway. 00:50:14.440 --> 00:50:18.389 HORVITZ: Tessa, bring us  home—bring us home! [LAUGHS] 00:50:18.389 --> 00:50:21.600 ALEXANIAN: Bring us home! Let's do it  faster next time. [LAUGHTER] Come talk   00:50:21.600 --> 00:50:26.000 to any of us if you're dealing with this  kind of stuff. You know, I think IBBIS,   00:50:26.000 --> 00:50:31.440 especially, we want to be a partner for  building those layers of defense and,   00:50:31.440 --> 00:50:36.960 you know, having ripped out our hair as a  collective over the past year and a half   00:50:36.960 --> 00:50:41.200 about the right process to follow here, I think  we all really hope it'll be faster next time. 00:50:42.640 --> 00:50:47.280 And I think, you know, the other thing I  would encourage is if you're an AI developer,   00:50:47.280 --> 00:50:49.440 I would encourage you to think about how your   00:50:49.440 --> 00:50:53.040 tool can strengthen screening and  strengthen recognition of threats. 00:50:53.680 --> 00:51:00.160 I know James and I have talked before about how,  you know, our Google search alerts each week send   00:51:00.160 --> 00:51:06.880 us dozens of cool AI bio papers, and it's more  like once a year or maybe once every six months,   00:51:06.880 --> 00:51:12.000 if we're lucky, that we get something that's  like applying AI bio to biosecurity. So,   00:51:12.000 --> 00:51:15.600 you know, if you're interested in these  threats, I think we'd love to see more   00:51:15.600 --> 00:51:20.960 work that's directly applied to facing these  threats using the most modern technology. 00:51:20.960 --> 00:51:22.880 HORVITZ: Well said. 00:51:22.880 --> 00:51:27.920 Well, Bruce, James, Tessa, thank you so much  for joining me today and for representing the   00:51:27.920 --> 00:51:33.680 many collaborators, both coauthors and  beyond, who made this project possible. 00:51:33.680 --> 00:51:38.480 It's been a true pleasure to work with you.  I'm so excited about what we've accomplished,   00:51:38.480 --> 00:51:42.240 the processes and the models that  we're now sharing with the world.   00:51:42.240 --> 00:51:46.480 And I'm deeply grateful for  the collective intelligence   00:51:46.480 --> 00:51:51.760 and dedication that really powered the effort  from the very beginning. So thanks again. 00:51:51.760 --> 00:51:52.368 [MUSIC] WITTMANN: Thanks, Eric. 00:51:52.369 --> 00:51:53.040 DIGGANS: Thank you. ALEXANIAN: Thank you. 00:51:53.040 --> 00:52:07.920 [MUSIC FADES]