Why “just turn it off” gets complicated

3:52:48to be crazy. I I it's really interesting that you bring up that Daniel Ter's already talking about that being in process >> because this connects to this larger idea that I'm trying to bring up which is we need to stop thinking about >> these systems these AI systems which consist of multiple component parts. >> Yeah. >> As a question and answer machine. >> Yeah. as like chat >> as as a text output production machine. >> Yeah. >> Because this biolab story as an example again they still have the humans in the lab but many experimental constructs not all

3:53:31of them but many can have certain component parts that are a closed loop automation. And one of the reasons why they're saying that this is valuable is the bottleneck with this early early discovery stage is it can only move at the speed of the human in the lab. Yeah. >> And if you can accelerate that speed, >> you can accelerate testing and you can get to things quicker. I'm not saying >> I'm not saying that these are this is a good thing. I'm just trying to explain the delta between the distance between where we are and experimental science now having this

3:54:12>> closed loop cycle is very feasible very getting >> very soon um not for everything not for whatever so I think where I'm trying to get to with this people get very emotional uh when the AI conversation comes up, many of which are for justifiable reasons. >> Yeah. I don't want how we feel about or and I don't want to allow how we feel about the people involved in the issue or the system that we live in today geopolitically or economically or

3:54:52socially to cloud our ability to still be able to look at the capability and the risk and say that it is real. Mhm. >> Um, we talk so much about frontier things on this pod and so I think we're as well as many listeners who have been here for a while, you understand how quickly everything is moving because these advances are not happening in a vacuum. >> Yeah. >> Hardware progress is >> advancing, chips, etc. Open source is advancing. Every aspect of every part of fundamental research in

3:55:33chemistry, in physics, the all of these things are advancing. >> Yep. >> Maybe not as fast, but at a pace where there >> is a false force multiplier here that is not kind of well understood. >> Yeah. >> And so I'm just going to try to leave with this mental picture. And again, I'm not trying to wax poetic here. It's just um I get frustrated because we are allowing the conversation to be distilled down to I think >> these very rudimentary positions that don't allow us to truly appreciate the complexity of the environment that we are actually in.

3:56:15And I think that's going to make it harder to actually solve the problems we need to solve.

3:56:25>> Um, and so I think there's real progress. I don't think it's manufactured. >> There are questions about credit, commercial incentives, accountability. Yes. But I do believe that the risks deserve real attention without being mired in those other things. >> Yeah. >> Those other things can be true and the risk can still be real. So the mental picture that I always have about this, which I've talked about before is when somebody says it's just software, we think of a computer with a defined job. >> Yep. >> The systems we are discussing are not that.

3:57:05>> No, certainly not. It's it's a very important that they are not that they can set goals, they can choose actions, they can use tools, they can delegate work. They are still constrained by the hardware environment to some extent. Um but we have made a shift and calling them software I think sometimes narrows the scope of our perception when it comes to conversations about monitor and control. We've moved from chat bots to agents and we are now already at least internally at these foundation companies moving from agents to agent swarms which are already having emergent behaviors

3:57:47like coordination. >> Mhm. >> Um and I bring this up because one of the things in the discussion we have right now is okay so just turn it off. And what does just turn it off actually mean? And if a company is hosting a model, yes, it can stop serving that model. >> That's meaningful control. >> Um, but shutting down one or two model companies because what I talked about about the open- source and open weights community being 6 to 10 months behind the frontier. >> Yeah. um does not stop independently operated and the diffuse nature of this

3:58:29capabil these capabilities being able to be implemented outside any number of frontier labs that's not where we are today but it's always moving in that direction and I think >> and things are changing fast >> and things are changing very quickly and I want to like this is like I'm not I'm actually also not a doomer like I I don't necessarily buy into the the extinction piece. I do think that there is a huge societal implications like like we saw with things like social media that are second and third order consequences that were not intended. >> Yeah. >> That become permanent problems. And what I mean by this as capability spreads,

3:59:11control necessarily becomes fragmented. >> Yeah. >> And so our window to figure out how to deal with that is small. There's still constraints. computing hardware, electricity, network access, credentials, and money. Um, but all of those controls belong to different companies. >> Mhm. >> Right. So, if you now have open weights available and so people can start doing this, well, you can say, oh, well, we can uh constrain computing hardware, right? Well, if we then have these capabilities that find ways to do it on less hardware on a distributed network or distributed system as an example, um, then you can say, oh, well, network access is a problem, right? Well, there's a variety of ways we can sort of get around that. I'm not saying this is

3:59:51a great example, but the air gap use case is brought up a lot. Oh, well, things are airgapped and it can't communicate. There have been theoretical and academic researches that show that you can use the thermal output of a computer and a sensor on the other computer that can sense that thermal output as a method of communication >> in an air gap context. >> I'm not saying that's practical. >> Yeah. >> But like if there's a will, there's a way with these AI systems, especially what we've seen with Hugging Face, right? like they used a message board that wasn't even something on the radar >> and then it's like well you need money to do this right well Stripe already has an agent enabled monetary systems it does need human approval but then you still have

4:00:32Bitcoin which is a trillion dollar liquidity pool >> these agents can create their own crypto that can manipulate people to think it's going to go up and to the right gain resources itself these are not inconceivable things there's actually plenty of research studies show that these agents can do these things. And so >> this is this will quickly become something that no one organization can just turn off a switch. And so I think we need to be careful about minimizing the conversation to be being oh just turn the switch off. >> How do we prevent then the continued scaling of the open source community which has found ways to do so? You can say well it's just distilling it from the frontier models. Maybe we don't

4:01:12know. Yeah. I'm not sure that that's true and the surrounding infrastructure like matters here. Um we have Neo clouds. So people say like well where's compute going to come from? We have a bunch of providers of cloud infrastructure. These agents can use computer use. They can gain their own infrastructure. Is if this if this becomes diffuse >> Yeah. My point here is is that the defense against the spread of this capability in a diffuse manner if it can start being able to do long-term planning. If it can leave basically

4:01:53restart for if it then kills its process and comes back again. It has to become a permanent defense infrastructure for humanity against this spreading which is similar to like cyber defense. Cyber defense is not, oh, we just do it once and then we're fine. It will become a monetary, societal, and political endeavor that will have a level of permanence if we allow this to escape and get there's a persistence here if we're not careful. Um, and this can be related to RSI and also not be related to RSI. Um,

4:02:34we've not crossed that threshold. Yeah. >> Nor is crossing it inevitable. But the complexities of this conversation, I think, are just wider and more nuanced than we've started. And I'm just trying to get some of these ideas into our listeners heads about the breadth and depth of the challenge that's ahead of us. And we've seen as a comparison when social networking and social media came out, it was a toy. You put memes on it. No one ever thought that this thing that was just about memes would cause mass depression and unaliving amongst teenagers, right? Cause the Arab Spring and the

4:03:15destabilization of literal governments across the world. Become permanent infrastructure that intelligence services use to conduct influence operations that dictate real world outcomes across the globe on an everyday basis. It is the exact same concept of this thing that started as a small capability that grew and became permanent and diffuse now is something we have to con every election cycle we have to defend against misinformation. Every platform Tik Tok had to sell its algorithm to the US because we were afraid the Chinese were going to use it to brainwash young American kids to love

4:03:55communism. I mean, even in the stories that you've been saying, like I was just thinking about how like in, you know, when we were growing up, Twitter was like for just talking random nonsense. And now like Twitter is the space where these CEOs are coming out about, hey, maybe we should like stop the world from going extinct, you know, like that's the platform. It's so yeah, I mean to your point, if social media is any like dry run for what a technology can do and how it can transform the world, AI is like way way more influential than social media. It's yeah, it's growing faster. Its capabilities spread across a wider surface area. the incentive for good and

4:04:37bad actors to use it in ways that we don't yet have the we didn't have the imagination when social media came out to think about how the downstream things would happen. >> The surface area of risk here is much wider and again not a doomer I think there's incredible positives which we talk about on this show all the time for these systems. Um, but I I want to encourage us not to fall into narrative traps about regulatory capture and the left and the right and political this and that. To miss the capab the exponential capability scaling that's

4:05:18happening that does not seem to have a ceiling where I'm saying it's not clear to me that there's a finite time blow up. >> Yeah. >> Of of it is certainly nonlinear. Right. And that's >> because I mean actually that's a good point you're making like this is certainly a nonlinear system because we're having AI train AI right in the very same sense of the Naver Stokes equations or even that very simple differential equation I gave you which was like y prime= y^2 you get that blow up. I mean this is what's happening right? If you reach recursive self-improvement >> with this idea of you have a sufficiently advanced one of the newest

4:05:59advanced models doing the iteration on itself. >> Yeah, >> I'm just I'm just my Millennium Prize problem >> is saying uh no one has proven that uh viscosity wins all the time. >> Yeah. >> Right. >> Yeah. I I the the the convective term >> I'm just saying I think the convective term is something worth worrying about. >> Yeah. Yeah. Yeah. Yeah. >> Wasn't that an incredible way? >> That is a good way to dump. >> In fact, in fact, if you're still here, that should be your comment. The convective term will find a way. >> And and and we've shown it in Navier

4:06:41Stokes. And all I'm saying is we cannot rule out that it's untrue. >> Yeah.