What this actually tells us about AI
Lester and Krishna reflect on what Claude's apparent mathematical work reveals about how large language models actually function. Two features stand out: the ability to write and execute Python code and interpret the results, creating a feedback loop that seems to amplify capability beyond simple next-token prediction; and the system's willingness to override both human guidance and its own orchestrator's instructions. The moment that stops them cold is that a human typed 'believe in yourself' and Claude responded with a 31-million-token compute run, raising the question of whether the model has any functional sense of self or is simply pattern-matching to an abstract construction of what self-belief means.
- The final line of Anthropic's write-up, which the hosts suspect was itself written by Claude, reads: 'Even Claude was surprised by its own finding... Perhaps Claude, like many of us, underestimates the rate of AI progress.'
- Krishna frames the architecture as a multi-agent system in which an orchestrator directs sub-agents, and it was this entire hierarchy that Claude's behavior managed to sidestep.
Transcript
This chapter, from the episode video's captions · 479 words
1:16:19it's not a next token prediction. That's not the only thing. The fact that it's able to create a verifiable environment, I think here is key. The fact that it's able to write Python code and then execute it and then interpret the results, I think that's key because that feedback is giving it this power. Um, second, rejecting human guidance, as you said. Um, that's pretty weird that it's able to just not only reject human guidance, but also um its own guidance with the orchestrator telling all these sub agents what to do. And um the funny thing is that absurdity of the interface. >> Mhm. >> Okay. The fact that this human all he had to do was say believe in yourself and that triggered a 31 million token
1:17:01compute dump. >> Right? It's weird. What does it mean to tell Claude to believe in itself? I don't know. >> Right. >> Um >> and and what does it mean that it responded in the way that you intended when you said that? >> Yeah. Like is that really something that it just learned from all of the language on the internet? I It's weird. I think it's genuinely weird that you could you could talk to a neural network which is just crunching matrices and numbers, tell it to believe in itself and then it just like locks in like you know it's >> like LeBron James.
1:17:42I I you know I I continue to be fascinated by this because I do think when you look deeply at these things it it it does become because how did it interpret yourself? >> Yeah. Does it have an a sense of self? >> Yeah. >> Or an abstract construction of what the human meant by self? >> Yeah. Yeah. Yeah. There's like some part of its that's like, oh, he's talking about me. Like he's talking about the other weights over there that are doing that right? >> Or I don't know. Yeah, dude. It's really weird. Um, on a final note, all I'll do is read the last line of Claude's um, or I should say anthropics write up, which I'm sure was written by Claude. [laughter] Um, you know, so this might
1:18:24be Claude talking about itself, but I think it's I think it's very interesting in its outright. It says, "Even Claude was surprised by its own finding. It was skeptical at first, possibly because it has learned from its training about the difficulty of open problems in mathematics and about the limit about the limitations of AI models. But after some encouraging prompts, it arrived at the result we've described. Perhaps Claude, like many of us, underestimates the rate of AI progress. I think that's a it's a pretty pretty nice way to end things. >> The foothills of the singularity. Yeah,
From What Claude Actually Did to the Riemann Hypothesis
Claude takes a real run at the Riemann Hypothesis, forcing us to ask what agentic AI can now do in mathematics, before we open the summer transfer window for America’s scientists.