AI Risk, Oversight and the Race Ahead

3:23:25which was the same day that OpenAI announced the Navier Stokes solution, this is where something that many listeners have probably already heard happened. 5:00 p.m. So OpenAI in the morning, they announce announced Navier Stokes at 5:00 p.m. on September 8th. Former anthrop now former anthropic researcher Jacob Coxin puts out the now famous tweet 160 million views. I resigned from Anthropic today. I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. What you just said, they are race. This is a very important sentence. This is a

3:24:06very important sentence. They are racing straight to self-improving super intelligence and gambling with our lives. More thoughts below that sentence. uh self-improving super intelligence which sometimes people understand it as um recurring self-improvement. >> Yeah. >> Is usually what these researchers are specifically identifying as where they view the nexus of the threat being. So when people have reacted to this, they'll go to their chatbt and say it can't do some function that I need it to

3:24:47do well. So, how is it going to be a danger to us? >> And the I think the point that a lot of the researchers are trying to explain is how we've gotten the Astra model that is currently working. Well, Anthropic just released Fable and Mythos and Opus 5.5 in the research process for how do we make the models better. They have both slowly started to integrate AI as a partner in the actual research process. M >> and so now the decisions being made about how do we make the next model better are incrementally being given to AI for portions and parts and the amount

3:25:28of portions and parts that AI is being given in the in the research process which has been exclusively for the smartest human beings we have that's right it's becoming more and more and more and so when people say recursive self-improvement or they talk about super intelligence what they're the underlying thing they're really talking about is the in either a majority of or the entirety of the actual process to go from model one to improving it to model two is entirely done now by some number of AI agents in a coordinated fashion. And the problem

3:26:09that arises when you start to have recursive self-improvement for such large portions of the actual research process is we as humans no longer we lose the ability to think about things like controllability, moniability and it can go in directions we could not even imagine. Right? And if we look at the hugging face example, who like it just becomes a runaway capability that we seed control over the potential of being able to control and monitor. It's insane. So RSI is a very important aspect of the threat that a lot of these researchers are referring to, not necessarily narrow AI context of it

3:26:52being applied in the way that people use it every day. And I bring that up because the follow-up tweet that often gets conflated with Jacob Coxin's initial tweet, which again, >> I'm going to bring it up again. He didn't give a percentage of human extinction. >> Yeah. >> Or anything. He just said they're acting responsibly. Yeah. >> And they're racing straight to RSI. Uh Evan uh Hubbinger Hub, Evan Hub, a different anthropic researcher who was still at the company, not not resigning, says um he puts his personal probability of AI killing all humans at greater than 10%. Um the actual tweet he said is Jacob is correct here. We really do

3:27:32earnestly believe that AI could kill all humans. I personally think it's greater than 10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for super intelligence and are clearly not on track to do so. He's he was currently an EP an employee at Anthropic when he tweeted that, >> which is nuts. >> He's basically saying like, "Hey, I'm doing this." You know, when he says, "We don't have a plan." It's like you >> you you buddy, >> you don't have a plan. >> And this is this is very >> You work What do you mean? You work there. This is >> I believe Anthropic is trying its best. You believe you are trying your best.

3:28:16>> What the hell? >> On se this is the same day Navier Stokes comes out September 9th. Coxin and this just blew up crazy. Coxin does a wired interview uh where he's asked, "Well, how is it going to kill us all?" And then he goes through some explanations of some of the parameters and the different things. And so this discussion starts spiraling. Everyone starts weighing in with their perspective. All of those groups that I talked about earlier that have different p vested interests in the outcome of this were all sharing their points of view. And then on September 12th, this is when we got from anthropic CEO Dario Amade his essay titled we must

3:28:56pace the frontier. And this essay cited this faster AI assisted AI development. again this idea that making better models is becoming more AI assisted >> it's a very important factor here not just humans making it better um and so he created this sort of policy regulatory global collaboration uh framework where he proposed embedded independent evaluators at all of the frontier model companies coordination amongst companies and democratic governments to create some sort of you know regulatory construction and ultimately global uh coordination

3:29:36particularly with China >> with a commitment that Anthropic would now embed evaluators unilaterally without anyone's patting himself on the back for saying we're going to start doing our job. >> Yeah. >> Right. Um >> and the proposal was to pace the capability development. Right. And again this is kind of getting in the weeds but I think these details are kind of important to understand how these systems work. So the constraint that exists on making these things better is compute. Yeah, >> the world is compute constrained. >> Yeah. >> And so each of these companies in their race to be the best and win has some finite not blowup amount of compute that they can apply to different things they

3:30:17want to do. >> Yeah. >> So they've for the most part optimized for rapid improvement of the models and then some amount of compute maybe goes to alignment and safety. And so when they say pacing the frontier, in part what they're saying in practice is we want to take our finite amount of amount of compute and instead of allocating only 5% to safety and alignment, we want to allocate 20% to safety and alignment. But by doing so, we're decreasing the amount of compute we can apply to making the models better, faster, stronger. And so it's it's internally it's a reallocation of

3:30:59compute so they can be doing their jobs. You know you understand what I'm trying to say. >> This is so stupid. >> But do you get like that's >> Yeah. Yeah. Yeah. So that's what they're saying in words >> in words. Yeah. >> Um as it relates to this and and so people have started to sort of try to parse this and what's so interesting is this is September 12th. Now this is 4 days after Navier Stokes, >> right? >> That so I I actually want to come back to this. So Amade put that tweet out at what time is this? He put that tweet out at 7:00 a.m. on September 12th. Okay. >> At 8:00 a.m. 1 hour later. >> Oh my god. >> Elon Musk, owner and CEO of SpaceX,

3:31:41which is now also XAI and they have their own AI stuff. An hour later said, "Daario is right." >> Amazing. >> It's all he tweeted with some caveats afterwards in a follow-up where he qualified what he meant by Daario is right. He says, you know, supports oversight beginning with peer review by competitors. Uh this, you know, he has his different ideas, but he's like, "Okay, Daario is right." That was one hour after Daario's post. I do think it's interesting the timing of the release of statements by the CEOs of all the AI companies that respond to Daario is almost a perfect reflection of like the companies they run and how they do things. >> And so Elon just does >> immediately. Yeah. >> Does everything immediately. That's his

3:32:22sort of uh MMO and how he does it. Then followed very swiftly at 9:30 a.m. by Sam Alman. Uh I agree with Daario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Uh hm, that's interesting. Committing to having independent evaluators with employee-like access is a great idea and we will do the same because anthropic said we'll unilaterally do it. Uh uh we'll have more to share soon, right? Okay, great. We're going to start doing our jobs. >> Yeah. >> So, with an hour in an hour and a half, within 2 and 1/2 hours, uh we have three of the major model companies saying we're going to do something. Okay, great. Now,

3:33:05same day 400 p.m. Sir Demis Hassabis >> ah Nobel Prize winner >> Nobel Prize winner kned uh the former uh CEO of Google DeepMind who he has actually left as of August 5th to become the chief scientist at Alphabet. Um supports Daario's proposal while saying the details still need to be worked on. You know, Dario's essay points towards the right path forward. The details need some working, but the direction is correct. This is why we put out our proposal for an industry. So, everyone's trying to say like, we're doing the work. We're doing the work. >> Okay, great. >> This is this has not happened, right?

3:33:45Where there's sort of a coalescing of all of the major model CEOs at the same time. >> That's on September 12th. September 13th, who's missing from the party? Uh well, we can talk about our lovely friend Satcha Nadella over at Microsoft who did a big deal with OpenAI and they were very early on it and then pulled back a little bit a little bit and he had as you can see different from the others a much longer nuanced Microsoft like response where he argued that we need to be careful about having any kind of system concentrate power amongst a small group of few. We need to make sure open models are incorporated into this.

3:34:27So much more nuanced thing. And oh, also check out uh our code of conduct. We have something too. We're not lagging behind, right? Day later. Okay, I promise this is the last one. >> September 12th, September 13th, there is another player in this space in the US that has said nothing. Put your comment in the comments if you think you know who it is. But not until September 15th, 3 days later, did we get Zuck, our our our boy Mark Zuckerberg, fink D on X, putting out his long explanation as it relates to all this kurfuffle. And his view was every lab has a responsibility and the

3:35:08incentive to move at the pace required to train its models safely and the ability to take its own actions to ensure that happens. Uh and what he basically was saying is we've delayed the release of our products. The most recent one being Muse, an agent type product for consumers, which you can now get access to on Instagram. He said, "We delayed it for months because it was not aligned and it was not safe. We didn't need anybody to tell us anything. We didn't need to demand the government come and help us and give us a binky because we're so incapable of making our products safe. We just did it." >> Yeah. So he's basically like, "Bro, like, >> yeah, just grow up. >> Just grow up." Right. >> This is in kind of in line with what

3:35:49David Sax has been saying, though. >> This is exactly This is exactly >> It's just like, I mean, you're responsible adults. Like, if you're if you think you're making a bomb, maybe stop and don't make the bomb. And this is where the all-in pod squad, the besties led by David Sachs, >> uh, have made this argument, which is, and this is where the narrative has arisen, which is these model companies are just trying to get the government to they're burning cash quickly, right? They're not profitable. Uh so they're creating this hype >> in order to force a narrative that the government needs to regulate this industry and create all of these

3:36:30requirements that are going to become cost prohibitive for startups to come into the space. >> Yeah. >> And do something. >> And like I mentioned earlier, multiple things can be true at the same time. Yes, there might be a regulatory capture uh benefit depending on how that regulatory infrastructure is defined because there's plenty of ways to say if your market cap or your revenues are above a certain amount these apply to you and if it's below this certain amount it doesn't apply to you and then it doesn't matter and there so it's very solvable by the construction. It is also true that David Saxs and the besties are all VCs and investors and have friends

3:37:13that have bets. >> Yeah. >> That might be benefited by not having that regulatory infrastructure. And so these are not people that >> are neutral players in this conversation. Again, it's valuable to talk about it, but look, you guys like you have a huge financial incentive for a certain outcome in ways that many other players do not. >> Now, not everybody in AI world uh was jumping on the bandwagon of, ooh, please come regulate us. uh former lead of AI research at Facebook Meta, who has since left, Yan Lun, uh responded uh to

3:37:54Daario's claims, uh saying, "Right, Daario was already claiming that GPT2 was too dangerous to open source back in 2019. I made fun of them then. Everyone should make fun of them now." >> Dude, he's such an animal. >> He He just not convinced. >> Yeah, >> not convinced. And so this is not a universal opinion about people who work at the frontier. There's a variety of uh differences of opinion. Um and I think you know again this gets back to this has created this dichconomy of either you think it's hype or you think it's going to lead to human level extinction. And again there's a huge gap in between these two things. But in the category of leading to potential human extinction. I

3:38:35do want to note another thing that happened on September 18th. This is 10 days after the Navier Stokes solution. This is insane. >> This is moving so quickly. >> My mind is so crazy, dude. >> This is moving so quickly. And I don't mean to keep mentioning the dates, but it's just it's on September 18th, an