The hosts discuss AI-driven discovery outside pure mathematics, starting with a chess analogy: you can be certain Magnus Carlson beats you, or that Stockfish beats Magnus, without knowing the exact winning moves, which the hosts use to argue that AI capability risk can be real even when tied up with commercial incentives. They cover OpenAI's new mathematics advisory group (hosted at the Institute for Advanced Study) formed after backlash over its Navier-Stokes claim, including its statement that the same internal model has resolved over 100 other unverified open problems. They then turn to Anthropic's newly announced biology lab, where Claude proposed hypotheses from literature that led human researchers to find a CRISPR-like enzyme sitting beside repetitive DNA, though its function remains unknown. The segment closes on speculation about fully automated wet labs removing humans from the experimental loop entirely.
- 01
The hosts note current AI systems have already been linked to real-world harms at smaller scale, citing cases of people experiencing psychosis or self-harm after interactions with chatbots.
- 02
OpenAI's advisory group includes mathematicians such as Ed Witten and Timothy Gowers, but explicitly excludes advice on the pace of OpenAI's internal math research.
- 03
Gowers has compared AI-driven math discovery to astronomical sky surveys replacing manual cataloging, but the hosts push back, arguing astronomy surveys remain scientist-run rather than corporate-driven the way AI math claims are.
- 04
Anthropic's bio lab experiments are described as BSL-1 or BSL-2 level, meaning they do not involve organisms considered highly dangerous to humans.
- 05
A researcher named Daniel Toker reportedly described labs working on brain organoids that are being pursued as fully automated, removing human hands from pipetting and sequencing steps.
2,094 words · auto-generated from the episode video
3:38:53article. >> And this is how long after Navier Stokes had been solved? >> 10 days. >> That's insane. >> 10 days. An article comes out saying that Anthropic is operating a lab that conducts biological experiments. And the reason that this is kind of interesting because as people have been trying to ask the question understandably particularly in the general public how how is this going to lead to extinction quote unquote? And so people have been forced to give these scenarios about how it leads to extinction. And so there are all of these, you know, what people might define as contrite examples like, oh, you know, takes control of a biolab and
3:39:33leaks a virus or it takes control of nuclear plants and forces shutdown or satellites and causes a Kesler effect and crashes the the communication systems or shuts down grids or it does any number one any number of these things. >> Yeah. in different places at different times and then also floods the information space such that it's difficult to even have human coordination to stop it. You could come up with all these examples. I think uh something I heard I think it's Nick Sorer's say that's I think a better way to think about the capability and threat conversation you know was this example of saying
3:40:13let's take chess right I can feel let's say you you are going to play a chess game against uh the best chess player in the world >> Magnus Carlson yeah >> Magnus Carlson let's Okay. I can feel very confident Magnus Carlson being this recursively self-improving AI. >> Mhm. >> And you just being >> me. >> You. >> Yeah. >> I can say very confidently that Magnus is going to win. >> Yes. >> That that's not No, there's not really a
3:40:53argument against that being true. >> Mhm. how he's going to win or what is the last move or what is the sequence of moves by which he wins may be harder to define but it's not hard to define this larger macro point that you will not beat him. >> Yeah. Yeah. And I mean, I could even take that analogy further and say that in a match between Magnus Carlson and Stockfish, which is the the latest sort of model to play chess, I can say very confidently that Stockfish is going to win, but maybe I won't know what the last chess move is going to be. Yeah.
3:41:37And so I I think I'm trying to create space where we can talk about there being a capability >> that is risky. And that doesn't mean that the model companies have a financial incentive. That doesn't mean that there's maybe regulatory capture. That doesn't mean that everyday people don't see a benefit. These things can all coexist, but we should not use the model companies having a financial incentive to effectively say that then means there is no capability risk. >> Yeah. >> And the capability risk does not have to be human level extinction because I'm sure if this starts killing 40, 50,
3:42:1860,000 people a year and then becomes a million, two people, two million people a year, it's not people are not going to be happy about that. I think something that's important to note is with current capabilities with these things unfortunately we're s seeing people being put into psychosis and unaliviving themselves. So we are already having deaths being contributed >> on a small scale because of interactions with these things >> and when we have things like wet labs being created which again is a kind of loaded term and they're only a BSL1 or BSL2 uh meaning they don't work with things that are a threat to humanity. >> Yeah. But come on. Okay. Um, so
3:43:00that's 10 days after Navier Stokes, which we just talked about how crazy it is. Now, on the 21st of September, OpenAI put out this advisory group on mathematics and artificial intelligence. And a very interesting point about this announcement that I want to reference is in the first paragraph they say on August 28th we began training a new internal model as we talked about earlier. In addition August 28th right now when we're recording this episode it's September 23rd going on 24th. >> Yeah
3:43:40>> less than a month ago. >> Yeah. In addition to resolving the Navier Stokes Millennium Prize Problem, CND, this model has now resolved more than 100 longstanding open problems across most areas of mathematics. These have not been independently verified. It will be interesting to see when they choose to put those out what that'll look like. But this advisory group that they've announced, they put it together because of all of the heat that we just talked about and the drama with their Navier Stokes result. So they've put
3:44:21this sort of advisory panel together to help them understand and figure out and work with the mathematics community on how to address this and deal with this. It's being hosted at the Institute of Advanced Study. It includes folks like Ed Whitten, Tim Gowers, Martin Herrer, other mass magicians. You can see the list of them in this post itself, but it's going to look at the advice from this advisory group around reviewing results, how to communicate their significance, um, and the general impact on, you know,
3:45:03research standards. But OpenAI explicitly excludes advice on pacing its internal mathematics progress from the group's remmit. Unsurprisingly, I promise I'm almost done. This is just crazy, dude. 100 problems. I wonder which ones they've got. And also like Tim Gowers, I mean Ed Whitten is kind of like out of the out of the game. Timothy Gowowers is a fields medalist who is like a proponent of AI and he he he's been talking about how oh like um you know AI finding mathematics is like how astronomers now find like
3:45:44galaxies and objects in the sky using sky surveys like they're not named after for example Messier who has the Messier catalog now it's just named after like NGC new galactic catalog because that's like the cat I mean there's similarities but there's also differences right because like the the the people in charge of the Sloan Digital Sky Survey or Vera Rubin are also scientists and researchers themselves. It's not like some like Carl Zeiss uh you know company that's like a big telescope manufacturer that's like trying to take over astronomy. This this here is a is like a community that is trying to I mean it's it's a corporation or a series of
3:46:25corporations that are trying to use mathematics as a leverage to talk about how great their technology is. That's not something that astronomy does when they discover new things. Like researchers are still involved in astronomy. It it Yeah. Anyways, that's what I mean obviously Timothy Gow is a great mathematician, fields medalist, blah blah blah, but I think he really missed that take when when he said that he was comparing like modern day astrophysics to what mathemat what what mathematics is going to become. I don't I don't think it's the same thing. I I it's a very good note and I'm just trying to provide a a zoom out breath of the ways
3:47:06in which this touches so many other things. Yeah. >> And I think uh the lack of depth and complexity and nuance in the conversations we're having around it. Um >> and clearly OpenAI did not expect the level of negative >> Yeah. >> reaction. And so this is a little bit of it's not backtracking, but it's a little bit of PR >> to try to, you know, because if they could have, they probably would have just done drops every day of like solved it, solved it, >> solved it, done. >> Um, but now they're realizing it's not in their >> best interest. The last item I'm just
3:47:48going to have in our timeline, which thankfully we waited to do this episode because all of these things have now happened. I mentioned that on September 18th, 10 days after Navar Stokes from OpenAI, the article, the alleged B bolab from Anthropic was in the works. >> Mhm. >> Well, conveniently on September 23rd, earlier today, Enthropic announces the first results from its new Bolab. We have up here now this promotional video that they put together.
3:48:29The ultimate result of what they discovered is this enzymes gene that sits beside a long stretch of repeating DNA and it resembles many aspects that are similar to crisper where every other kind of sequence they've looked like that has this cut and replace kind of functionality related to it. And while the function is still not known and has not yet demonstrated its capability as a gene editing tool, it has all of these interesting hallmarks of a potential now pathway of discovery that could be usable. And so
3:49:12the way in which they implement this is they gave Claude a bunch of data and scientific literature to propose hypothes hypothesis >> and then and candidate systems to give to real human anthropic biologists. Mhm. >> They reviewed those ideas and then performed their the experiments that they thought had the mo best potential based on the ideas from Claude >> and they're now basically trying to say hey this is a first step we did it's one of its hypothesis found an interesting
3:49:52structure and now we are going to continue down the process of scientific discovery to see if we can now get functional benefits out of this structural discovery. Um, >> it's amazing >> and it's it's really so and what they're trying to point to and again this is kind of in contrast >> to the Navier Stokes solution where Anthopic is trying to show like look we're enabling scientists as a tool. We're not just trying to take the claim. I'm not saying it's a perfect >> Yeah. >> solution but you see >> Yeah. And the and the other thing is I mean in their in their announcement on their tweet they say we don't yet
3:50:32understand what this system does but only a handful of known systems share its features. That's very much a fundamental science type of research right where it's like um we're just curious. I mean their PR is a lot better than open AI. I have to say >> they they they they are doing something. Yeah. Um and then D >> and this is exactly how Crisper was founded, right? It's like, oh, this repeating stretch of DNA. A shout out to, you know, we've got a great episode on Crisper. >> Yes. >> That um hopefully we'll be alive enough that you can you can watch it. >> Things are happening very quickly. >> Yeah, it's incredible. It's absolutely
3:51:12incredible. If you are a longtime listener of the show, you've heard us talk about AI enabled fundamental research before. very narrow context, very well- definfined, not blackbox physics like our buddy the brain scientist who created a non-b blackbox version of this. >> Actually, speaking of the brain scientist, >> um, and Daniel Toker, >> um, he was telling me about labs that are completely fully automated, >> really wet labs, biological wet labs. He works in organoids and like creating like mini brains on a petri dish. And there there are efforts now to completely automate wet labs. So now imagine you could completely take the
3:51:53human out of the loop of biological research. You could have claude or open AI any of these frontier models read through literature as you said propose hypothesis and then have an a fully automated lab go through do the pipeetting do the gel electrofpharesis and you know all of the sequencing and everything. If all of that is automated, like where exactly is the human here, right? Are we gonna now see nature papers that are by Claude and Open AAI, right? Like cuz right now it's like these are math papers that are by Claude and Open AAI because it makes it's just like all like thinking in some sense and like churning through lean code trying to figure out
3:52:35if something works. the day that experimental science becomes fully automated and like there's a paper by Claude about you know the next generation of crisper that's going