Roman Concrete, Brain "Cognitive Legos," DeepSeek, and Econophysics

Intro — Season 2 momentum + what we’re covering

0:00Hello internet. This is your captain speaking, Lester Narre, joined as always by my co-host and our resident PhD, newly nicknamed some physics nerd Krishna Chowdery, courtesy of Young Jamie. >> Yes. How's it going? >> Um, it's been an interesting week. >> It's been it's been pretty incredible, right? >> I I I got a DM and it was a screenshot of your face. >> Yeah. >> And Joe Rogan's face. >> Yeah. And And I thought it was AI. [laughter] I thought it was just some deep fake and it was it took us a minute to find it. So, kudos first and foremost. We're back for season 2. We're super excited. Uh this week has been a little bit crazy

0:41because uh Young Jamie, the producer over at the Joe Rogan Experience, decided to reference our Nuno Lorero uh Christmas special episode in the middle of a UFO segment, no less. >> Yeah. >> Uh and it's that was very unexpected. >> Yeah, it it was pretty incredible. and he called us some physics nerds. >> Yeah. >> Point of order. Uh I am no physics nerd. Uh >> and it was a it was a it was a compliment to me. >> It was a compliment. It was a compliment. >> It was a compliment. You're an honorary physics nerd. I feel like you've uh you've learned a lot. >> Yeah. I mean I've learned a lot. So >> So look, if I can uh quote Navier Stokes equation off off RIP, you know, it's not it's not too bad. Exactly. Um

1:22>> we're going to dive in. We have a great episode today. We have a new segment that we're going to try out for uh in between one of our stories and we're going to talk about a variety of cool topics. We're going to start off with Roman concrete and then pivot over to our favorite institution, Princeton. Go Tigers for a new cognitive Lego. >> Yes. >> Which I love Legos. We can see from the Lego that's behind us here. >> Yes. But this is the [clears throat] Legos that make up our thoughts and our abstract ideas in our head. I think it's a very cool story and it's very cool research. >> Minority Report inbound. There was another deepseek moment >> uh took over the internet. They did a

2:03January's back to back. I think R1 was January last year. Yep. >> And now some new something about constrained >> manifolds. >> Manifold constraints. Very complicated. And we will end our story with our former classmates's favorite finance broology. It's our first economics story. >> Yeah. >> Uh that'll be interesting. And as always, we're going to get into the weeds of the research papers, the methods, the experimental design. You might learn something today because this is from [music] first principle.

2:44>> [music]

2:52[music] >> Okay. Our first story is about Roman

Story 1 begins — Roman concrete’s durability mystery

2:57concrete. Have you ever been to Rome? >> I have not. I was just in Italy, but I did not get the chance to partake in the life of the Roman. Yeah, Rome is honestly one of my favorite cities that I've ever been to because the amount of history that's there and the continuous civilization, you know, it's it's been around for millennia now. And you can see buildings that are literally 2,000 years old that I can scarce believe they built 2,000 years ago. It's actually like pretty incredible. And all of that is because of something called Roman concrete. >> Okay. Um, there's been a breakthrough

Pompeii: a construction site frozen in time

3:37because a recent excavation in Pompei has actually led us to figure out how they made that Roman concrete. And it's been elusive to even material scientists in the modern day how the Romans made that and how they could build such buildings that last 2,000 years and are still used today. Okay. It's pretty incredible. I think I think I heard about that on one of the tours or oh the concrete is >> Yeah. Yeah. I mean it's really it's like really quite something if you think about it. So there there was this thing called the Roman architectural revolution. Okay. It's this widespread use of Roman architecture.

4:17Previously unused forms of architecture. They used to have arches, vaults, domes. Before the Romans, the Greeks never made this kind of stuff. Okay. But because of concrete we can actually do it. This is the pont dug guard in France. What we just saw was the semic-ircular arches that were in the theater of Marcelus. These this is all by Augustus, the emperor Augustus, who is um >> yeah, he's the guy who took over right after Caesar got stabbed. Right. >> Right. Um >> A2 Brutus. >> Um yeah, A2 Brutus was August uh was Caesar. And then um Augustus said uh you know uh friends, Romans, countrymen, lend me your ears. But Caesar but Brutus is an honorable man. Right. [laughter]

4:58Great. Great. One of my favorite Shakespeare plays. So, they're they're building all this crazy crazy stuff, right? 40 AD. This is the time of Jesus. The crown jewel is really the Pantheon, which was built by Emperor Hadrien in 126 AD. That building is still intact and people can go inside and like there's like no damage to it. It's pretty insane. 2,000 years after it's built, the Pantheon's dome is still the largest unreinforced concrete dome in the world. >> That's today. >> Today, that's actually pretty incredible, especially given the scale of architecture projects that are happening. >> Yeah. Yeah. Yeah. Yeah. It's It's pretty

5:39insane. I think the Super Dome was like coming close, but then like Hurricane Katrina happened. But this is a single dome. It's a concrete dome with a hole on on the top, right? It's massive >> and it's just still around entirely made out of concrete. We have our next photo where it shows you can see it from the Roman skyline. That's the concrete dome, right? And it's still it's that's 2,000 years old. >> That's so crazy. >> Yeah, that dome is 2,000 years old and it's still alive to this day. And all of this has to do with the fact that the Romans invented Roman concrete. Okay.

Self-healing in seawater — why Roman structures get stronger

6:16It's and and we still can't figure out how they did it. >> Okay. >> But we're getting close. >> That that to me that that that that's how do we not know? >> Yeah. I mean, and we're we're going to find out why it's why it's so difficult. [laughter] Right. In 2017, there was a paper um in the American Minologist from the University of Utah and they actually showed that this when this concrete got in touch with marine ecosystems, so with salt water, it would actually strengthen itself. It was like a smart material where it was like repairing itself if there was a crack in the concrete. Unlike modern concrete where the thing just like, you know, you got to decommission it, you got to tear it down, build a new bridge. This stuff was

6:57actually self-reinforcing. >> Okay. So, that was in 2017 already. We've got like a smart material that's

Vitruvius, lime, and what “mixing” really means

7:04better than modern day concrete. Okay. So, the question is, >> how do you do that? >> Yeah. Yeah. Because we see it in superhero movies when they get the thingy or or uh the Foundation, the Apple TV show. Uh, one of the, spoiler alert, uh, one of the characters is this like sentient robot that's lived on forever and has like self-healing kind of >> and the Yeah, these Romans are doing it 2,000 years ago, right? So, it's like it's like what's going on? Okay, so we want to figure out how it is. The first clue comes from a guy named Vuvius. He actually wrote the first architecture book in the world back in 30 BC. It's called Day Architectura. um and he described a way of mixing the

7:46concrete. So, what you do is you take something called quick lime, which is just calcium oxide. That's something that you get when you um take limestone, you heat it up. Limestone is calcium carbonate. That's like a sedimentary rock that you get from, you know, millions of years of sedimentation from all of these ancient sea creatures with like sea shells. >> Okay? >> You pack up a bunch of seaells, you get limestone. You heat it up, you get quick lime, which is Cao, calcium oxide. >> Okay. >> Okay. And when you combine that in excess water over months, you get some stable form of lime putty. And that's the process that he described in his book, Architectura.

8:28>> Okay. >> Yeah. Okay. So the defects that they found whenever whenever you look at this Roman concrete in like super detail, you'll find these millimeter grain to centimeter grain defects of lime. It's like full lime. So it's like not fully mixed in. >> Okay. Okay. And it was just thought that like all of this all of this lime that's like not fully mixed in is is kind of a defect. Okay. Then in 2023, a paper from MIT comes out. Okay. This is from the Mesaich lab and this describes a new form of mixing. This is hot mixing. So in the previous one in Vuvius in his document that's 2,000

9:09years old. I mean the guy was there presumably. So he's describing how he saw the concrete being made. He said it's cold mixing which is >> this calcium oxide. >> Yeah. Which which is you get the you get the calcium oxide, you mix it with water and then you mix the other stuff. >> Okay. What the MIT guys are saying is no. Actually, what we do is we get the lime class. We get we get that lime. We don't mix it with water first. We mix it with volcanic ash and all of these other ingredients and then we put in the concrete. And that order >> is very important. Order operations. Yes. Because I told you in chemistry, chemistry, this is basically materials chemistry, right? And like it's like cooking. The order of operations is very

9:50very important. Okay. So, what the MIT 2023 paper says is that we're going to do hot mixing, which is mix the lime with the volcanic ash, mix it together, and then put in the water. Okay? And then these chunks of lime,

“Hot mixing” vs. “cold mixing” — the modern rediscovery

10:06they're going to be the reason why the Roman concrete actually can self-heal. >> Okay. >> Okay. It's by design. Okay. >> It's not a defect. Okay. >> Okay. >> Okay. >> So, the central problem, right? >> Who's right? Yeah. Right. >> Right. Hot or cold? >> Like like who is right? Who's the who's the guy who actually can do it? Um like is is it the hot mixing or is it the cold mixing? >> Right. >> Can I take a guess? >> Yeah. >> Actually, no. Go ahead. >> It's the MIT guys. [laughter] It turns out It turns out the original author >> 2,000 years ago [laughter]

10:47>> I think got it wrong. But there might be a good reason for it. and we'll get to that at the at the end. >> Okay. >> Okay. >> So, >> the paper comes out um last month. >> Mhm. >> In nature communications, >> photo number 12.

Pompeii evidence: dry pre-mix piles + intact quicklime

11:01>> And this is the paper, an unfinished Pompean construction site. >> Okay. >> Okay. That's the key. >> Okay. >> Pompei is the key. >> Okay. >> Okay. I went to Pompei um I think it was two years ago. I actually that's I did go I did go to Pompe go to Pompei. Isn't it incredible? Yeah. Pompei is um >> an intact Roman city. It's >> that has been buried under a volcanic eruption. Mount Vuvius erupted in 79 AD >> and it buried an entire volcanic city in moments in just like you know within within

11:44within a few hours. And when you say city, it's, you know, particularly for the time, it is so [snorts] unbelievably large. >> Yeah. >> And organized and choreographed. It's like hard. >> It's It was a really surreal and bizarre feeling being like, "Oh, that street goes all the way over there and this one goes all the way over here." And it's a grid system. It's really impressive. >> Yeah. And there's a shopping district and there's like intact restaurants >> and you can you can you can tell what the function was because it looks just like >> today. Yeah. Yeah. There's a mall. >> Yeah. >> You know, it's insane when you think

12:25about it. Right. >> It is. >> And the advantage of studying a site like Pompei >> Yeah. >> is that >> what you've got is a city that was 2,000 years old. um one of the you know beach cities of the Roman Empire and it's frozen in time >> and just like every other city >> there's going to be construction sites. >> Okay. >> Okay. You see you see what's happening. So if there's a construction site you can go and look Yep. >> and see how they were mixing the concrete. >> Yes. >> That's the key. >> Yes. We can see how the sausage was made. >> Exactly. >> Uh because much like in all as all time periods in human history, we're always building something. >> Yeah.

13:05>> Exactly. and and you know that like they're renovating a building or they're building a new building or something like that. So they found a construction site and that's the focus of this paper. >> Okay. Got it. >> Okay. They found an active construction site >> and what they found was the dry premixed piles. >> So not lime that's been sitting in water and cold mixing, >> but actually lime that's been put and mixed with volcanic ash. >> Okay. >> And then it's waiting to be mixed with water. >> Ah, okay. Okay. Got it. Got it. Got it. Got it. >> And the reason why this matters chemically, right? We want to talk about like what what's what's the difference between the hot mixing and the cold mixing. >> Yeah. Like I you know, and I can kind of

13:46maybe into it a little bit, but it seems to be really fundamental. >> Yes, it is. It is very fundamental. So the idea is whenever water reacts with um calcium oxide, it gives off a lot of heat. >> Okay. Okay. That's the hot mixing part. Okay. Okay. Now, if you do it the way Vuvius said all those years ago in that book, then you take the lime, you mix it with the water, and because it takes such a long time, the the heat is just going to dissipate into the water. And so that's the cold mixing. >> If you do it when it's hot, right, you do it when it's hot, then you've got a mixture of the lime and the

14:26volcanic ash that you're mixing together, that's going to give off a lot of heat. there's gonna it's going to reach temperatures nearing 200 degrees C. Okay? And in that chemical reaction, what's going to end up happening is it's gonna sort of plaster everything together and that's actually going to preserve chunks of lime before they fully dissolve. >> I see. >> Into the rest of the mixture. >> I see. I see. So there there's sort of this this timing this this timing function where there's a window where it can solidify prior to distribution. >> Yes. >> But only at certain temperatures. >> Got it. >> Exactly. Yeah. And and because it's

15:07happening so fast, the the chunks of lime don't have enough time to really diffuse and become part of a homogeneous mixture. And so having these grains of lime in your concrete is actually a good idea. I see. >> Okay. >> Okay. Okay. >> And the reason for that >> I was gonna say Okay. Yes. >> Yeah. And and the reason for that we're gonna we're gonna sort of get into a little bit later, right? >> That's actually key to the the self um healing process. >> This kind of random distribution of this lime inside the mixture that solidifies >> in this case at high temperature prior to it all kind of getting distributed

15:48like into a muck. allow has some derivative effects that create uh favorable outcomes. Exactly. From a construction perspective. >> Yeah. Yeah. From a construction perspective. Right. So they found these dry mix piles and they're like, "Okay, this is this is great. That means that, you know, we were kind of right in that there's lime mixed with volcanic ash and it's dry. It's not in the form of the the concrete." Okay, let's let's look at some of the the smoking gun microructure that they found. Okay. When they looked at the unfinished walls and then they looked at the raw material, what you see is this really brittle micro structure where you're getting like the the you're

16:28getting chunks of lime, you're getting microscopic bits of volcanic ash and the two are exactly the same. Meaning the raw material and the unfinished walls, >> the walls that have just been created, >> right? Unfinished means there's no plaster, which means it was just done. >> Yes. Those have the same material properties. >> So, so this is again helping to verify the the the hypothesis that it is unfinished. >> Yes. >> Cuz when you look at it when it's really small, the structure of it matches because if it was finished, the fundamental that microruct Yeah. Well, no, hold on. The unfinished the fact that it's an unfinished wall means that it's very recently been made.

17:09>> Gotcha. >> Okay. Yes. And and so what that means is that that construction happened from this pile. >> I see what you're saying. You see what I'm saying? Yeah. It didn't have because because over time because >> Yeah. Because finished means that they put a plaster on it and then like now this the wall is smooth. Like if you've been to inside the Pompei buildings, there's like paintings and stuff like that. This is a new wall, right? And the raw material is right there. So you can you can connect those two and be like this >> was going to go become that. >> Yeah. I got you. >> You know what I mean? and it's right next to each other. And because the micro structure is exactly the same, you can say that actually, yeah, they did dry mix and then and then do the and then do the hot mixing and then and then make that wall.

17:49>> Right. That's the idea. >> Got it. >> Okay. [snorts] Um the other the other

Isotopes, microscopy, and validating the mechanism

17:54really cool technique that I thought they used was something called stable mass stable isotope mask spectroscopy. Okay. Here's the idea. So, you know, in nature you've got isotopes. Isotopes are elements that have different numbers of neutrons versus protons. And so they're going to be heavier. Like for example, oxygen um 18 versus oxygen 16. Same chemical properties, but it's just heavier. >> Okay? >> Which means if something gets hot, right, the the heavier oxygen is going to move slower. And so in any reaction, there's going to be more lighter oxygen

18:35that's gonna do the reaction versus the heavier stuff, right? >> Because it's moving faster. The lighter oxygen is moving faster. So you can actually calculate you you can you can do an experiment and you can figure out from mass spectroscopy how much of each isotope is in the substance that you're looking at. >> Got it? >> Does that make sense? >> It does. It does. >> Right. And so that's what they did. What they looked at was how much of carbon 13 there is and how much oxygen 18 there is. And what they found was that the the the slope of of these lines if you look at photo 16 right >> it's characteristic of that water limited high temperature fractionation.

19:16>> Oh I see what you're saying here. >> So so the the gray triangles >> are what they found >> at the site in Pompei. And what you're looking at the the red box is where most of that those gray triangles are. And that's exactly what would happen if you premixed. >> And then you and then you put water. >> That's that quick lime. >> Yeah. That's that quick lime. Exactly. And so the slake lime samples, that's the cold mixing. That's the stuff that you want to use for finishing because it's smooth and you can plaster it onto the wall and then you can, you know, put mosaics and like paintings and the ornate stuff. But if you want the structure, you want you want the concrete that actually self-heals, you

19:56actually want to do it with the hot stuff. >> This is so like and so it's interesting like there there was also this understanding of the dual use at these different at these different >> That's what's insane. That that's really cool. And and here's here's actually like what's what's sort of happening with the the quick lime. Okay. And I I haven't gotten into how how exactly it would work, right? like how how would why would these chunks of lime peel concrete? >> Mhm. >> Right. Here's what's happening. So suppose you've got concrete and you've got a crack. >> Okay. >> Now you've got a crack in the concrete.

How lime clasts heal cracks (and why it’s not magic)

20:32>> There's going to be water that goes in >> Mhm. >> into this concrete. If it's normal concrete, that water is going to push out more and push out more and degrade the whole thing. But if you have chunks of lime, calcium oxide, right, that's going to dissolve into the concrete and fill, sorry, dissolve with the water and fill the crack >> and become a kind of putty >> that like seals that crack. >> Yeah. >> Over time, >> it's a it's like an extended release. >> Exactly. So, you have these like pellets You have these pellets of like glue that get activated. >> Right. Only when >> only when there's a crack and the water goes in and then it seals that crack. That's so what's so amazing about that is,

21:14you know, they clearly had intuited that. >> That's insane. >> And what could be perceived as imperfection. >> Mhm. >> Initially actually had these uh unintended, you know, you consequences that actually made the underlying structure more robust. And just like the the open-mindedness to think about stuff and try things Yeah. to get to that cuz in this day and age we like oh no it's not the mixture is >> and and that and that's what we assumed for the longest time we assumed that like this Roman mixture was just like not good >> good >> you know but no there it might have been by design >> right >> that they did this and and it's it's

21:57dude it's it's so insane and this this actually makes sense right because Roman inspired concrete this could be a huge thing for us environmentally because Roman inspired concrete this is the Pantheon again, it's lasted for thousands of years. So even though in the process of creating this concrete, it might use more CO2 than let's say the more environmentally friendly concrete, the biggest carbon footprint when building something or the biggest carbon footprint of a building is the process of building it. >> Mhm. >> Not the materials themselves. >> That makes Yeah. If you build it once and it lasts for 2,000 years, >> that building is the greenest building

22:40>> on the [snorts] planet. Like I I would argue that the Pantheon, which is still in use, >> is like the most environmentally friendly building because it was built once and then it's been used for 2,000 years. >> Right. Right. >> Right. >> Yeah. Yeah. Yeah. Yeah. It it it's it's really hard to because also the scale of it, right? And I mean it rained for 3 days here. >> Mhm. >> And everything's falling apart. >> Oh my gosh. Yeah. [laughter] Yeah. Yeah. Yeah. Yeah. It was and and there's potholes everywhere. It's just it's just the worst. Right. So it would be really cool if we could actually rep start replicating this ancient technology and then and then start using it. And so,

23:21um, Admirich, who's the who's the lead author of this study, he actually does have a startup called DMAT

DMAT and modern “Roman-inspired” low-carbon concrete

23:29that is trying to make sustainable and durable concrete solutions inspired by ancient Roman concrete. >> That's >> okay. So, he's he's using this knowledge that he's gaining from all the research that he's doing on ancient Rome to commercialize eco concrete. And they've actually got pilot projects in the US and in Europe right now. >> Shout out Admir. Uh well done. Yeah. >> Uh well done when everyone thought it was nothing to see here. >> Yeah. Yeah. And he he kept at it, right? And he proved Vuvius wrong. Now there's a there might be a reason good reason that Vuvius was wrong. Okay. And it's because that Vuvius actually wrote his um architecture textbook in like 30 BC.

24:12But the architectural revolution that we talk about the pantheon pontugard and all that that happened about a hundred years after. So it could be that in those hundred years 100 years is a long time right even in ancient Rome. So they could have they could have actually figured out this new technique of hot mixing >> after he wrote that textbook >> and you know given the given how fragile >> any textbooks are maybe somebody else wrote another textbook but it's just been lost to the ages right so that's probably one of the reasons the other reason the other reason could be that it was just like translated wrong from like ancient >> you know Roman whatever Latin I guess they were using Um, so there's a lot of

24:55reasons, but this is a pretty good like nail in the coffin in in the cold mixing vituvian >> vuvian >> theory >> theory. Yeah. Right. Right. And >> like we're pretty sure we're right that it is hot mixing and and the the the group at MIT has actually replicated the the the architecture, right? >> And they've actually started a whole company with this new type of concrete. So it's clearly working. I was going to say the way the way you know something is working is when the uh >> the capital markets get involved. That's when you know it's uh >> it's transcendent. >> That's fascinating. I remember walking around in Pompei and just being like this is crazy. >> This is Yeah, dude. I was so impressed

25:36with Pompei >> and this it this makes sense. I know there's always a lot of memes about, you know, the sort of story about the Romans made concrete that we can't figure out. And it one of the things about this story that's perseverance is really such a fundamental aspect of like scientific research because one like you're constantly battling just to maintain funding to continue to do the to search and like people give up after a while if you can't meet some kind of quarterly >> annual whatever metric that is independent of just the curiosity of it. Yeah. Um, so kudos to the team. I It's

The Rundown segment begins

26:10It's always hard for us to give um big ups to MIT. But >> yeah, >> I think US college rankings had them at number one and us at number three. I think this is fake news. >> That's that's totally fake news. >> That is This is nonsense. >> Yeah, >> this is absolutely >> I'd like I'd like to know their >> absolute nonsense. Tiger Nation number one. We're going to get to Tiger Nation. Speaking of which, in our second story. But before we do so, we want to do our new segment, the rundown, where we're just going to hit on a couple of other recent breaking research stories that have come out uh this week that we're not going to go deep into, but are still fascinating nonetheless. And we are going to start with uh the

26:51moon rehearsal. >> Mhm.

Rundown 1 — Artemis II “moon rehearsal”

26:53>> Which happened on December 20th in preparation for this is the sort of Artemis 2 final dress rehearsal. Uh NASA successfully completed a full launch day simulation for the first crude lunar mission in half a century. So if you don't think we went to the moon before, >> we at least might go to the moon now. Let's see. Uh this is a collaboration between both NASA and the Canadian Space Agency. >> Yep. >> Uh we have astronauts Christina Cotch, Victor Glover, Jeremy Hansen, and Reed Weissman. Uh it's going to be a 10day mission, which is super exciting. And they're taking off, I believe it is February 5th, so it's coming soon. Yeah,

27:33it's going to be great. It's basically like Apollo 8. They're not landing on the moon. Um, but they're not landing on the moon, but they're actually going to be going around the moon. Yep. >> And then coming right back. And one of the coolest things about this is um it's it's going to be the farthest that humans have ever been from planet Earth. >> Really? >> Yes. Because even in even in the Apollo missions, you know, they went around the moon, but the the guy who stayed in the lunar capsule or sorry, in the command module rather than the two guys who went down in the lunar module, the guy who stayed in the command module, Michael Collins in Apollo 11, for example, was

28:15pretty close to the surface of the moon on the other side. He was on the far side on his own. um with these guys. These guys are actually overshooting and they're going to go really far out >> and then catch catch a wider orbit. >> Yeah. Yeah. So So they're going to be the farthest humans ever from Earth and it's going to be I I'm I'm really excited to see the photos that they get. It would be cool to see like are they going to be so far that they can get the moon and the Earth >> in the same shot? >> It's the the pale blue dot V2. >> That would be insane. That would be really really cool >> cuz we don't really have a photo like that of like >> you know our our family like earth and the moon together like that.

28:56>> Actually we do. That's not true. We do have that from um these there's this Japanese satellite that's quite far away right in between the sun and the and the earth. Okay. >> And every once in a while when the moon goes in front of the earth it gets a photo. >> Doesn't count. There's no people in there. >> Yeah. Exactly. This would be like a a photo taken by a human being. >> Us. That would be >> not our not the fruit of our loins, which is technology. >> Yeah. [laughter] >> Yeah. >> So, we we'll be on the lookout for Artemis 2. We actually have uh Artemis behind us >> uh over one of our shoulders. Oh, over the other shoulder. >> Oh, this it's [laughter] it's this camera now. We're still getting used to the three camera setup here, but it's

Rundown 2 — IDEGs / LLM-guided evolution for discovery

29:34right here. >> That thing took me not a reasonable amount of time to build. Uh so, that's our story number one in the rundown. Number two is the cellular cleanup crew. This came out uh January 7th. It's about ID digs, IDGs, which are deleting diseasecausing proteins in nature chemistry. So scientists have found a way to supercharge our cells natural recycling systems to completely destroy cancer shielding enzymes. And we've talked about uh uh T-Rex and and the whole immunity uh immune system and the immune response in our past episodes in season 1. Some really big deep dives

30:16that are great on this. That's what kind of caught my eye on this one. >> Yeah. >> So we have Max Plank Institute involved in this led by George Winter. And the idea here is that most drugs today act as inhibitors that just like block, right? They they they they block the site. They don't really, you know, activate. But >> this this is now recognizing and destroying the targets entirely. >> Yeah. And it's using the body's own machinery. That's I think what's key, >> right? >> It's like it's like it's basically priming these proteins and not not actually like taking care of the problem themselves, but like tagging them for the body's own >> um trash collectors, these E3 liases to

30:59do it themselves. And it's always good to just like have the body do it, >> right? Because you the body has a whole bunch of dynamics. You talk about biology, there's all these variables. >> Yeah. >> There's a lot of variables. >> Yeah. It's if you if you can help the body rather than just do the task yourself, it's always better, >> right? >> So the the analogy was if if a uh standard drugs are like putting a lock on a door to keep a villain inside, IDEGs are calling in a demolition crew to remove the entire room, making sure the threat is gone for good. Yep. >> Again, this is all happening like this like like this is this week. >> This is this week.

Rundown 3 — SleepFM and sleep stimulation

31:32Third rundown the chat GBT moment but for sleep Jan 6 sleep FM this new thing out of Stanford medicine >> that is diagnosing 130 diseases while you sleep uh it's hard to avoid AI in stories all the time because it kind of just is what it is but this story is fascinating FM was trained on 600,000 hours of physiological data to be able to spot subtle mismatches between your brain your heart and your breathing signals. Yeah. >> And from that, it can then make predictions about a variety of diseases. >> Yeah. And what really got to me was it only needs to analyze one night of sleep data from you

32:13>> to figure something out, right? It's it's trained. It's so it's a pre-trained AI kind of like the pre-trained transformer models that we use in LLM and things like that. But so this is an AI that's been pre-trained with, as you said, 600,000 hours of physiological data. But then all it needs to classify you is just one night of data and then it'll tell you one of the 130 diseases that you could have. It's pretty insane. We it being able to read biological data through like these like sort of derivative signals >> uh in such a short limited amount of like you need such a limited amount >> Yeah. Yeah. You could just like go and go, you know, to a facility that will monitor all of these different things

32:55while you sleep for one night and then that's it. >> Then you got it. Yeah, >> unbelievable access to information. What will we do about it? Tune in next week.

Rundown 4 — silica metalenses and chip-scale optics

33:04>> Story number four. It's now an optics and nanotech story. This was uh today. It actually came out. The silica breakthrough. High-tech lenses for common glass. This is out of Harvard. Frederrico Kapaso. So, traditionally flat metal lenses require this exotic and expensive materials to bend light. You know, we know as you get smaller in scale, things become more expensive. We see this with chips and all these other things. And so they've sort of perfected this single clean path of light without having to you like they've perfected this ability to do what used to take massive scale and massive cost with basically off-the-shelf glass.

33:45>> Yeah. Yeah. It's it's pretty insane. I mean, it's it's going to be way cheaper, right? Massive scale, lower cost,

Rundown 5 — imaging stomata: watching plants “breathe”

33:52>> and glass. We use glass in a whole a whole number of applications. >> Yeah. And one of the key things for me that I really liked about this story was that you can use all of the engineering that we've developed to make chips >> and silicon semiconductor industry stuff we can now use to make this glass. Right? That's one of the reasons why it's so cheap is because we can leverage all of this infrastructure that we've already built for all of our computing and now we're making glass out of silicon. >> It's always great when we don't have to rebuild our industrial base in order to be able to enable and access new technology. >> It's like discovering you can build a high performance jet engine with the standard parts from a local hardware

34:33store, >> which is fascinating. Our last quick rundown story of the day before we get back into our main stories is this ability to watch plants breathing really really really well. This is an environmental science January 7th stomata insight a real time window into crop survival. So the idea behind this is people are trying to improve how we do certain aspects of agriculture. the better we can monitor and see stuff, the better we can do things to sort of impact outcomes from a a crop perspective. >> Yeah, it's more it's like a methods paper that is like, you know, I now now

35:13that we have a way to measure >> the stomata opening and closing on the on the bottom of a leaf. we can figure out what are the genes that actually control these things and then well can I you know crisper these genes to then make a droughtresistant crop that doesn't open when it's you know too dry things like that >> this is a new imaging system that lets science watch the microscopic mouths of plants as they breed in high definition out of University of Illinois Urbana Champagne uh Joseph D Crawford and Andrew DB Leaky um very cool I love plants I love the idea of like you Having plants around is always super fun and fantastic. And that's just a little brief rundown. We can't always cover

35:54every story, but we're trying to illustrate there's so much happening. There's so much going on >> that does actually translate into real world. It's not all just abstract. >> It's not mathematicians kind of going back and forth. >> Exactly. >> About god knows what. >> But now, my favorite, another story about the good old Princeton Tigers. >> Yes. >> The best. This is actually from uh the Princeton Neuroscience Institute which is um the building where I did my senior thesis back in the day. Um I was I was a

Story 2 begins — Princeton “cognitive LEGO” compositionality

36:28part of the physics department but my senior thesis I wanted to I I was really interested in biological physics right >> and so there was um there was a professor there who was interested in physics and how how to sort of use physical intuition to talk about the brain. I was particularly focusing on the retina. This is talking about much higher order abstraction um task demand like thought and and and

Why compositionality matters (the brain as a reusable toolbox)

36:55really abstract things. But there is a very nice thread of physics and mathematics in this story and it is going to get kind of technical but we're in for the ride, right? Like that's that's why you guys are here. >> Yes. >> So it's it's it's really cool. Okay, the core problem is something called compositionality. Okay, it's a key in biological intelligence and it's something that AI is actually not very good at. >> Okay, and something that the authors used was, you know, if you already know how to bake bread, you can use this ability to bake a cake. >> Why? Because you can actually take parts

37:38of your learning when you baked bread, you know, how to how to mix stuff, how to turn on an oven, how an oven works, and you can take those models together and then figure out, okay, with a cake, it's like not that much different. find. There's like a little bit of frosting. There's like all this like other stuff. But you can take the Legos, the cognitive Legos that you've built when baking bread and then use that with some new stuff to figure out the ability to bake a cake from scratch. >> Okay. >> AI is not very good at this. >> Okay. >> Okay. like a AI's like in in terms of this

38:19generalized intelligence that we kind of are after >> whatever that means >> right whatever that means >> no clear definition but I know I'm >> you know what I mean like it's like it's it's it's good at certain tasks because you've trained it on certain tasks it's seen the data for those certain tasks right but as soon as you sort of bring it out of its comfort zone the performance decreases by a lot okay And for a really long time, it's been kind of a dilemma. This question, how can a neural network, right, that's defined by neurons connected to each other, so you've got these synaptic connections, you want it to be stable enough so that

39:00it doesn't forget old skills. >> Right. >> Right. But you also want it to be plastic enough. We want it to be on the move enough that it can actually rapidly assemble new skills. Okay? So, it's this dilemma between stability and plasticity. Plasticity meaning the ability to change. >> And we as as humans have really mastered that tight rope. >> We really have, right? We have >> better than every any other thing we can really see. >> Yeah. I mean, we have um millions of years of evolution to thank for that. But um in terms of AI, when whenever you have AI, there's this problem called catastrophic forgetting. Okay? And that's the idea when you you learn a new

39:42task and when you're learning a new task, it totally obliterates your ability to do some old task. So in this in this case, you've got a dog cat classifier and then you train it to, you know, figure out how birds look and then all of a sudden it can't figure out what a dog and a cat look. [laughter] Now this is an exaggeration because nowadays we have learning models that that can do much better. But at the end of the day this idea of catastrophic forgetting is still very much a problem. Right? If you want to expand to just generalized ability to solve tasks. >> Okay. >> Now in cognitive science we have another problem which is called the binding problem. And that's the idea of how does the brain dynamically bind different

40:23attributes to the same thing without requiring a single neuron for all of the specific stuff. And here's what I mean over there. You've got a a person who's looking at a rolling ball. The ball is red and it's moving to the right. >> Mhm. >> Right now, there's three attributes here that I can immediately see. There's motion. It's moving to the right. There's color. It's red. And there's form. It's a ball and it's rolling. All right. Now, the brain doesn't have a single neuron for red rolling ball to the right. Okay? Because that would be insane, >> right? >> Okay. I don't have like a single neuron

41:03for every little object and every little idea. What the brain does is somehow combine all of these ideas and bind them together into a single idea of a rolling ball that's moving to the right. Okay. >> Okay. And that's called a binding problem. >> Okay. >> Okay. It's it's a problem in neuroscience. And we don't quite know mechanistically inside the brain how these neurons are actually doing it. >> Okay. We we can you can talk about it like oh it's you know there's an idea of a red but at the end of the day like it's like what is the neuron doing cuz these are just cells in my brain >> and and it's an inc it is incredible because it's it's a highly efficient

41:44highly uh sustainable architectural system. >> Yes. Yeah. >> And like it would >> that can learn anything. >> It would be good to know how it works. >> Yes. >> For a whole number of reasons. >> Yes. Exactly. And that's what the key discovery of this paper that came out of nature is. Okay. What what they figured out is the neural basis at a systems neuroscience level of these cognitive Legos. Okay. And they used something called lowdimensional shared neural subspaces. We're going to get into what that is, but there's a lot of math involved and some very cool physics intuition, which is why I really love this paper because it reminds me of like sort of why I got into like physics and neuroscience and the beauty of that in

42:26the first place. >> So, what we're what we're really doing is we're solving the problem of this binding problem. >> Yes. >> Using very cool mathematics >> and it might actually give us insights into how to train AI better. Which makes sense because ultimately if we understand how to solve the binding problem like that's immediately going to be architecture you want to implement into any kind of artificial system that you want to replicate human intelligence. >> Yeah. Exactly. And AI is at the end of the day for a long time it's been inspired by um human cognition. So there's a reason why they're called neural networks. Right? >> Okay. So let's talk about some mathematics now. Okay. Here we go. So

The mathematical framing — tasks on low-D manifolds

43:10imagine you've got a bunch of neurons. Okay, >> we've got about I think 80 billion neurons. We're not going to talk about that many neurons, but what you can think about when we think about what is a brain state. Okay. >> Okay. >> Let's let's first talk about what is a brain state. A brain state would be a description of all of your neurons and which ones are active and which ones are not active. Okay? >> For example, right? So like a zero one like a map of zero. >> Yeah, that's one way of thinking about it. Another way would be like um what's the firing rate of each of these neurons, right? And um is it it? Yeah. But at the end of the day, it's something like is this neuron active or not? And then how is it connected to all

43:51the other stuff? But the the state of the brain at any given point, like right now when I'm doing this podcast, there's certain neurons that are turning on and off, right? and my brain is moving from one state to the other as I speak, as I go through what I'm trying to talk about and things like that. Okay? And given we've got so many neurons, we run into this curse of dimensionality, >> right? >> And that means that if you've got a lot of neurons,

Manifolds, neural subspaces, and “shared structure”

44:22your brain state lives in a highdimensional space. Here's what that means. Yeah. >> Okay. So suppose I only have three neurons. >> Mhm. >> Okay. Imagine a brain with only three neurons. That brain is going to live in a 3D space because the X-axis is going to tell you how neuron 1 is behaving. The Y-axis is going to tell you how neuron 2 and the Z neuron 3. So where I am here >> at any given moment [clears throat] is going to tell me what this three neuron brain is doing. >> Now imagine a thousand neurons. I've got a thousand different axes, right? And each state in this thousand dimensional

45:04space is a brain state. Okay. The curse of dimensionality is that if there's unstructured data all over the place, then the amount of data that is required to figure out where I am is going to be like ridiculous. >> Yeah. >> Okay. So, clearly the brain doesn't do that. And we know that it doesn't do that because >> like more of a brute force approach. >> Yeah. It it it would be like just a really terrible way to do any kind of learning anything like that right highly inefficient. It would take like longer than the >> lifespan of the organism to actually like get any learning done, get any cognition done. Okay. So instead what the brain does is correlate these neurons. >> Okay. >> Okay. Because neurons are connected to

45:45each other. So if one of them fires then it's going to then it's going to influence its neighbors to fire and so on and so forth. So the neural activity is actually constrained in what's called a lowdimensional manifold. A manifold you can think of as a surface. And here

Curse of dimensionality (and why the brain avoids it)

45:59you can think let's go back to our analogy of the three neuron brain. Okay, you've got a three neuron brain. Neuron 1, neuron 2, neuron 3. But let's say neuron 3 is actually correlated to how neuron 1 and neuron 2 do. Okay. >> Okay. So, my dimensionality is actually not three dimensions because if I know what neuron 1 and neuron 2 are doing, I immediately know what neuron 3 is doing because these two, let's say, are input to them. >> And so, neuron 3's activity is on a plane, right? And so, instead of three dimensions, I'm really only talking about two dimensions, right? and my my brain states are confined to this

46:40manifold in a higher dimensional space. The manifold being this lower dimensional 2D structure. The idea is you can there because of the correlation that that you know exists you can abstract the complexity to a a simpler structure because if ball goes up like if neuron one goes up if you know when neuron one goes up neuron 2 goes right >> you you don't you can now use that as a >> I know that information >> information right and that becomes sort of like >> I can rule out all of this >> other stuff that otherwise would have been in the probabilities okay yeah >> exactly yeah you see what I'm saying so Now you're constraining your your brain states to a lowdimensional manifold in

47:21this very highdimensional space. So even though you have a thousand neurons that you're dealing with, maybe the brain actually only lives on a much lower dimensional surface in this thousand dimensional space just like the three brain the three neuron brain was only living in a 2D surface. M you could even imagine a one 1D surface where it's like a line, >> right? And so all three of them are connected in some way and there's no there's no independence. It's just like all you have to do is figure out where you are on the line rather than where you are in 3D space. >> Okay. So the central insight here is that if we have two different cognitive variables, you remember we're talking about the binding problem, right? That's what we're trying to solve. We had the

48:01red and we had the ball and we had the motion. These are three different things >> that we don't store in individual neurons. >> Exactly. Right. So, we've got to figure out a way for the brain as a state >> to store this information. >> Okay. So, the way we can do this is if we have a giant highdimensional space, but we can constrain ourselves to manifolds. What if there's two different manifolds for the two different variables that we're trying to store? >> Okay. >> Okay. >> Yes. That's the idea. The key idea is you can have in your let's let's go back to our three uh three neuron brain. You can have orthogonal planes. You can have

48:43planes that are perpendicular to each other. And I've got I've got a photo to show this. So, you know, we can have a color plane >> which is one plane. >> And then I can have a shape plane, >> let's say. >> And that's that's the second plane or a motion plane >> and that's the second plane. So now >> here's what here here's what we would do. >> Your brain is somewhere in 3D, right? In this 3D space. But >> what a downstream circuit can do is look at all of these neurons and then figure out what is the shadow of my brain state on one plane. That's going to tell me the color. >> Mhm. >> The shadow of my brain state on the other plane, that's going to tell me the >> shape.

49:23>> Shape. Yeah. >> You see? So I can keep track of a bunch of different information. by constraining myself along these >> the intersections of different manifolds. >> That's the idea. >> That's f because the idea is that like for for lack of a better way to put it, it's I'm still groing, but like the the you have a you have a high a very high dimensional space of potential potentialities. >> Yeah. But you have a very like acute space whether it's on a single manifold or at the intersection of the manifolds which is actually what you care about. And that surface area that that that the amount of information or surface is like

50:05small is fundamentally smaller and shorter to traverse than navigating through the entirety of that larger multi-dimensional. >> Yeah. Yeah. It's kind of like you can't you can't like fully have unstructured just like brain space. Right. >> Right. like your brain states have to have to adhere >> conform to some >> conform to some order in order to actually like learn and like like get ideas and like keep track of ideas, right? So what what you could do is you could have a brain state that's in this high dimensional space and then you can have another brain circuit that's somewhere else >> that is looking at this brain state and then what it does is calculate okay where along this plane am I? >> Yeah. And then another the color circuit

50:46is going to be like actually I care about this other plane. >> So where along this other plane am I? >> Right. And then and then the binding problem >> is kind of solved. >> Solved. Right. Right. Right. Because you don't you you you you now have an ability to um store those the the the color the shape and the motion >> along these with the same number of neurons. >> With the same number of neurons. I don't I don't need individual neurons for red for blue for this that right and I don't need I don't need an individual neuron for red ball red square red star blue ball square blue star right I I can now

51:26with the same number of neurons all I have to do is just calculate where I am on this plane and this plane >> yes >> you see what I'm saying >> yeah I do I do >> very cool >> it's yes please continue sorry but yes yes >> okay so it's cool it's cool to think about >> yes >> but um what what truly impressed me about this paper is how they actually went about proving this. Right. Exactly. >> Because this is this is quite >> like very abstract. >> Yeah. >> Right. So like how how are we actually doing this? >> Prove it. >> Yeah. Exactly. [laughter] Um so they used um two adult rius monkeys. The rius monkeys are chosen because the prefrontal cortex the the the neoortex of a rius monkey is very similar to a human um neoortex. Okay. So

52:07you know for it's better than for example a lot of the other model organisms that we use like rodents because this brain can be involved with rule switching abstract categorization. So it's very close that way and we can actually try to probe these higher order abstract ideas. Right? >> And the stimuli that the subjects were responding to were of two kinds. Okay? There's shape and then there's color. The shape that they were responding to is a bunny versus a T. Okay. And the bunny ears became a T. >> Okay. Got it. >> Okay. So, so that's the sort of shape axis

52:47>> that we're trying to discern. Right. And then on the other hand, we have the color axis which is green versus red. >> And one of thing, one of the things that was that they were very careful about is that these axes are circular. >> They don't have edges. >> Okay. So you smoothly go from a T to a bunny back to a T and from green to red back to green. Okay. The reason why this is important is because you don't have any edge effects, right? Like if you if you have like a square, let's say, then there might be some artifact from being in the corner here because there's nowhere to go over on this side. So your your brain state might be doing something weird, right? But here in this case, you've got a circle and then

53:27you've got another circle. There's no edges. >> So every point in the circle is equivalent. >> Yes. >> Right. topology of this manifold, right? It's a Clifford Taurus because it's a circle >> which is a circle like this way >> and then a circle around the donut and then like through the you know there's there's two circles in a donut. So So you're constraining your stimulus >> to a Taurus. >> Okay. So presumably the brain should also have some kind of representation of this Taurus. And the task that they're doing is the following. Okay. So you've got a stimulus. The subject looks at the

54:09stimulus and then he does a sakad which is the eyes go in one particular direction based on whether it's a tea or a bunny or if it's green or if it's red. Okay. So the subjects are um trained to categorize shape in which case it's a bunny or a T. And they respond in one axis. So the axis being like you know upper left, lower right.

Designing compositional tasks (how they test the idea)

54:34>> That's going to be the shape axis. >> Mhm. >> And then there's going to be another task which is color. And they're going to do the opposite direction which is upper right, lower left. Okay, that's key. So we we need to understand that. And task design is everything. Okay. In a lot of these systems, neuroscience, behavior and abstraction experiments, I remember when I was in my PhD, like we used to spend entire lab meetings just discussing the task >> because the task design actually lets us figure out >> how to probe these very weird questions that we're asking, right? >> Cuz once we have the task, then we can figure out how to analyze what the neuron was doing in this task versus

55:15this task and so on and so forth, right? So the task is as follows. There's a shape axis which is let's say this way. >> And then there's a color axis which is this way. >> Okay. [clears throat] And then what they have to do is they have to switch the axes. >> Okay. So before the shape axis was upper left, lower right. The color axis was upper right lower left. And then now what they have to do is they have to say actually >> I'm going to respond to color but using the shape axis. Okay. Wait. Uh, okay. >> Okay. >> So, >> let's go through this again. >> Okay. >> Okay. >> Yeah. Yeah. Yeah. >> With shape, >> if I see the shape one way, I'm going to go to the lower right or the upper

55:56right. >> Mhm. >> If I see the color, I'm going to go to the upper left or lower left. >> But now in the third task, I'm mixing the two. >> Mhm. >> Okay. So, I I I'm looking at color, but I'm actually responding with the motion of the shape. >> Okay. >> Okay. Yes. >> Here's the here's the key hypothesis.

The core hypothesis — composition by shared neural subspaces

56:18Okay. If this cognitive Legos is correct, >> yes, >> then the brain should learn that mixing task, >> but it shouldn't learn it as a brand new problem. >> It should be just >> it should be a cognitive Lego shape task from the motion side and a cognitive Lego from the color task. >> Combining them together, >> the neural representations of those two. Yes. >> Should be resilient. >> Okay. Like it it does it's not ephemeral. It's not When you say resilient, how do you mean? >> What I mean [clears throat] by resilient is the same neurons >> that were responding to color are now going to still be responding to color.

56:59The same neurons that were responding to this axis are still going to be responding to this axis. But the the subject is still is going to be doing a completely new task, >> right? But using the same old the same old neurons. >> Yes. you using the same sort of procedure, right? The color processing module >> developed for one task >> and the the response the axis motor response for another task. It's going to just snap them together like Legos. >> Literally like Legos. Yeah. And and then that gives it the ability to then do the derivation without ascribing more neurons to do. >> Exactly. Yeah. Yeah. So, it's it's it's learning something, but it's actually it's it's it's able to learn because

57:41it's able to snap >> older older learned habits together, but >> at the neural level, the neurons are now working together. >> Yes. Yes. Yes. So, yes, the the idea is if if you Okay, I'm going to pause. This one This one is This one is >> This one's weird. >> This one's weird, >> dude. These systems neuroscience is is is really cool to think about that way. >> I just I Let's pause and and and really think about this >> because I know we have a lot to cover but I'm really trying to the point is right you have these you have these a neuron has a learning about whether it's shape or color in this context on these axes. >> Yeah.

58:21>> And let's say uh as an analogy the color is a red yellow a red Lego brick. Yeah. >> And the shape is a blue Lego brick. And so rather than saying when you you rather than having a new yellow Lego brick, you can just take those two put them together now and they are now able to get to this al this reversed response task that you're talking about >> this like combined >> combined this com this like combinator >> and the and the key is that we're combining two different modalities here. We're we're combining it's it's >> I see it's it's >> responding to color.

59:02>> Mhm. Right. >> But the response is with the shape response. >> Mhm. >> It's it's using the the the the axis that it's using it used to use for shape. Yes. >> It's a totally different task. >> But it is able to reuse an old a previously paved path. >> Mhm. >> For a different uh use case without losing the ability to still do the old thing. >> Yeah. >> And also not losing the ability to continue to do the new thing. >> Yes. Exactly. And so and one of the things they did was they randomized like which task they were doing and they found that the the you know the performance didn't decrease at all right which and that's that's something that like bi biological brains do all the time but what the key thing is what they

59:44can do is they can actually do simultaneous single unit recordings of five different brain regions at the same time. Ah so this is okay >> and this is where the neurons come in right so we can we can now do electrophysiology which is literally put in electrodes into the brain and then listen to thousands of neurons at the same time I think it's on the order of a thousand in this study from five different brain regions the parietal the lateral prefrontal cortex the frontal eye fields >> the temporal cortex the striatam right so this is where their sensory motor information reinforcement learning um preoter information. There's a executive hub which is the prefrontal

1:00:26cortex which is really the task rule. It's trying to figure out which task am I on in the first place, right? And that is what is really cool, okay, is the fact that we could use all of these things. And so how do how do we actually make sure that like how do we show that these cognitive Legos are actually working, >> right? Like how do you demonstrate existence? >> Yes. How do you demonstrate that there's these manifolds and all this other stuff? Here's what they did. Okay. So, you get a classifier that is trained on color. Okay. Okay. In task in in the in

The classifier test: can models decode the “built” task?

1:00:59the color task. >> Mhm. >> And what you do is you say, okay, now in this new task where it's trying to find color, but the motor response is completely different. When you say when you say motor response >> meaning in in the first task the color response was upper left lower right. >> Oh I I get >> but my motor response now in the second task is same color but now I'm going in the opposite axis. Yes. >> Right. >> If I train a classifier on the on the color task >> and I want to decode the color meaning I look at the neurons and I try to guess >> was the color red or blue right? >> No way. >> And I don't retrain the classifier. I

1:01:39just I just now test it on on the second task >> which is which is a totally different motor thing. >> It can still guess the color. >> No way. >> You see this is the key [clears throat] for me. Yeah. Yeah. >> Now now it clicked. Right. This result is very profound because it means that the geometry of the representation in this highdimensional space is stable. Right. >> That's actually No, that no that is and the way they got to it's they're clever. That's that's I love the I love these like neuroscience. >> So clever, >> you know. Yeah. Because because that's what that means. If you've trained a classifier here and then now >> this classifier is not trained on this second task. But because the neural

1:02:20representation is stable, I can take all of the learning I've done here and just apply it here. And I can guess if it was looking at green or red or whatever, >> which means that underlying manifold structure >> is the same. >> Is the like is a is it is it is a >> it's like a real thing. >> It's a thing. It's like a real mathematical thing in this highdimensional space. >> Wow. >> Very cool. Right. >> This is this one. This one. >> And they did the same thing for the shape. They did the same thing for the direction of the motor and all that other stuff. >> That's like a really big deal. >> Mhm. >> Yeah. I thought I thought it was very very cool. >> This is why everyone was rushing to get out their manifold stories early this year cuz it seems to be a confluence of >> Yeah, it is. It is kind of cool because

1:03:02the next story we're going to talk about is is about manifolds as well, right? >> Oh my god. >> Um, one one of the other cool things that they did was so you know if like they have a compression they actually what they found was >> Sorry. [laughter] >> Yeah, >> this is so it's cool. >> It's so cool.

Compression index & measuring “compositional efficiency”

1:03:17>> It's cool. They they had also this thing called a compression index. What they found was like whatever the task is not not yet sorry but like whatever the task is relevant for like suppose it was paying attention to color. Yes. then the shape manifold would actually like shrink. >> Okay. >> Okay. Like if you look at the neurons that were doing the shape stuff, they wouldn't they wouldn't be as active anymore >> because because the the the the subject is paying attention to the color first >> and then and then so so even the the the manifolds themselves are dynamic in that sense, right? The neurons that are in this plane are now a little bit shut off because they they're not as important right now >> anymore or right right now, >> you know, in in this particular task. >> Yes. I mean it's kind of like muscles,

1:03:59right? If you're not like you're not using them, they kind of get a little smaller, but but they're still kind of >> Yeah. But this is Yeah. And but this is like, you know, on the order of minutes that they're like responding because because the subject knows which task is happening >> happening in real time and then subsequently which one to then prioritize or have active in a more active state. >> Yeah. >> Very cool. what >> it's very cool to see just how how our brain has like >> yes >> done this you know millions of years of evolution and um we're getting very close to solving like stuff like the binding problem which is a very big deal you know >> yes >> and this has a lot of implications for

1:04:40artificial intelligence >> I mean because because catastrophic forgetting is very much a thing right so um there was actually this there's a new way of trying to figure out >> how to um not do catastrophic forgetting. So if we go to photo 15, right, there's something called orthogonal subspace learning, which is the idea that if I want to train on one task, task A, and then I also want to train on task B, what I can do is I can make the two subspaces orthogonal inside my neural network. And now when I do back propagation, which we're going to get into later, but it's just basically a way to train the model and change the weights of the model, the the way that

1:05:24the model is going to train for task A and the way that the model is going to train for task B are going to be orthogonal. So they're not going to interfere with each other. >> And you can enforce that constraint. >> I get >> like I'm I'm if I'm on task A, I'm only going to move along this plane >> in my parameter space. And if [snorts] I'm trying to do task B, I'm only going to move along this plane in my parameter space, right? And that way I don't forget how I did task A, right? If I now all of a sudden start trying to learn task B, >> it's it's putting in uh roads and traffic lights into the pathways of uh h how >> yeah, >> the processes proceed in a way that they don't interfere with each other.

1:06:05>> This is so cool. >> So yeah, it's um it's it's really cool. I mean, there's this new there's this new AI architecture called orthogonal low rank adaptation. O Lara or Oura it very recently came out. I think it was last year. >> Mhm. >> Where they're trying to do do this >> because the orthogonal the the orthogonal idea this like orthogonal subspace learning conceptually that maps to the same thing you were just talking about earlier about when you have plane this plane and the like that's the orthogonal that's what it it's literally and they're applying it now in the context of training >> but it >> we are we're able to experimentally see this >> see this in that's that is >> and like straight up confirm it.

1:06:46>> Right. Right. Yeah, >> that is very good. >> I thought it I thought it was a very cool play. >> I mean, it's hard to beat Princeton. Yeah. At anything. I mean, maybe maybe football. You might get us in football >> like all the time, >> but um when it comes to being the leaders in a variety of hard sciences. My My brain is currently having a lot of orthogonal subspace. >> Yeah. Yeah. Every time we get on this podcast, I have to do that. a lot of orthogonal subspace learning [laughter] but it's it's very cool and I think um you know in terms of so this is not just for um fundamental science right you might be asking okay like

1:07:26>> cool guys >> cool guys right [laughter] but you know cognitive inflexibility is um one of the symptoms of psychiatric disorders like schizophrenia and ASD right and you know >> given this kind of fundamental research it could be that that inflexibility comes from the failure here to orthogonally separate and compress [clears throat] subspaces in your head. So you can imagine like um non-invasive neurom modulation therapies that would like orthogonalize subspaces like make it more perpendicular. >> You have these headbands on that would yeah that would like that do the thing but but the point is there's a big

1:08:07>> I mean and that's that's far out but like you got you got to get to this kind of fundamental research. >> You have to understand that the manifold is there. [laughter] >> Yeah.

“Physics of meaning” — what this suggests about cognition

1:08:15Yeah. And I I think what fundamentally what I really liked about this paper is because like it it provides like a physics of meaning. >> Mhm. >> Right. It's like meaning is literally like a geometry in your brain. Like your brain is >> living in this highdimensional subspace >> and there's like a geometry underlying it >> that gives stuff meaning. >> I thought that was very cool. It's sort of the whole this is why math is uh discovered not invented. >> Mhm. It's even in our brain, >> right? >> Yeah. Very cool. >> Fascinating. This is our our story number two. Um out of Princeton

1:08:55neuroscience. I I just I I'm really struggling. I'm normally not like this. That that story really kind of I'm going

Story 3 begins — DeepSeek’s scaling moment

1:09:02to have to gro that one for a minute. But uh I have a good followup for it now because we are going to go into story number three which is another deepseek moment. Jan 1, they came into 2020 >> January 1st dude >> just like all guns blazing. >> Guns blazing. >> If it's all over the news um especially if you're in tech, etc. and all these other spaces. >> Yeah. >> But there's there's something about manifold constraints and they broke the scaling laws and everyone's hair on fire. How are they doing it? Ref, somebody please. >> Well, they're they're trying to beat scaling laws, right? Because um they're GPU poor at the end of the day.

1:09:42>> Mhm.

Scaling laws + the stability bottleneck in deep nets

1:09:43>> Um scaling laws are very effectively more compute, more parameters, better performance. >> If you have bigger, better stuff, you spend a lot of money. All more chips, bigger, better, faster chips means you can get to the next stage of >> Exactly. of >> whatever you're trying to scale up. >> Yeah. Yeah. Exactly. But there is a caveat, right? Which is at massive scale, you've got like hundreds of billions of parameters. I think now we're getting to trillions of parameters. You're going to hit something called a stability limit. Okay? And in deep neural networks, it's all about signal propagation, right? Which is like there's a signal of did I get something right or did I get something wrong? And as that propagates

1:10:24through my network, it's very chaotic, right? And it can lead to amplification, vanishing gradient problem, exploding gradient problem. These are stuff that we're going to talk about. So there's some kind of limit, right? There might be some kind of like thermodynamic limit to how much we can do. And the solution

Manifold-constrained hyperconnections (mHC) overview

1:10:44is what deepseek is purporting to have found. Okay, it's something called manifold constrained hyperconnections. It's a new way to train models more efficiently and actually like straight up better. Okay, but it's a very small hack that they're doing and it's actually quite simple. >> The hack that they're the the little hack that they did is like really simple but >> apparently it's a very big deal. >> Okay. >> Okay. So, let's get into it. >> Okay. >> Neural networks. First, let's just talk about neural networks real quick. Okay. Neural networks are a giant mathematical function that basically it's fancy linear algebra. Okay, this is um a

1:11:27transformer architecture from the very famous paper attention is all you need, >> right? >> Sorry, I repeated after him. I almost got through the whole episode without doing it. >> It's hard to unlearn 15 years of friendship. Please. >> Yeah. Yeah. Yeah. Well, so all of this is showing all of these arrows are basically numbers going from one thing to the other and then each block is effectively a matrix. You're multiplying matrices, then you add stuff, then you multiply a matrix, then you do a threshold, then you add stuff. It's a giant mathematical function at the end of the day. And each of the building blocks of these are something called an artificial neuron. Okay? And an artificial neuron is effectively kind of

1:12:10like a biological neuron in that it sums up its inputs and then if that input is above a certain threshold, well depending on, you know, the activation function, but let's just say it's above a certain threshold, then it lets it through. If it's below a certain threshold, then it doesn't let anything through. So that's that nonlinearity part. Okay? [clears throat] >> Mhm. >> And when we train a network, we've got these billions of parameters, billions of neurons that are all connected together. And what we want to do is something called back propagation. So you've got a network with weights which are how these neurons are connected. Um

Backprop intuition — signal paths through deep systems

1:12:44what you do is you initialize the network with a bunch of random stuff. Usually it's like random weights. You can get more complicated that way. Um what it's going to do is it's going to have a guess at an answer. It's going to compare its guess to the real thing. Compute a loss, >> which is how different was my guess from the real thing. and then propagate that back through the network and change the weights such that the next time it's going to be a bit closer to the correct answer. That's what back propagation is. >> Literally, the movie Tenant is a movie about humans back propagating in like real life. It's like I just made that connection in my head right now, but

1:13:24that's literally what they were doing. It's funny. >> That's so funny [laughter] actually. Yeah. Um, so that that that's effectively what back propagation is. And and really what you're doing is like the chain rule in calculus where you take the change here and you multiply it. You take a derivative here and you multiply it to the next layer to the next layer to the next layer and that's how you calculate how much you want to change your weights by. Okay. It's a product of these derivatives, >> right? >> Okay. Um something called a Jacobian matrix. We don't have to we don't have to get into what this Jacobian matrix is. But here is the key thing. Okay. Effectively, whenever you're doing back propagation, the amount that the weights

1:14:05here change is based on a product of all of these numbers. >> Right?

Vanishing gradients and Jacobian products

1:14:10>> Now, if the numbers that you're multiplying are a bit less than one, >> let's say 0.9. >> Mhm. >> But there's 100 layers, 0.9 * 0.9* 0.9 100 times is something like 0.00003. This is the vanishing gradient problem. Interesting. Okay. The idea is the the stuff that's changing, the weights that are going to change if I do this very naively, >> yes, >> is only going to be towards the output, >> okay, >> of my of my neural network. All of the earlier layers that I have like, you know, it goes input and then there's something something something all the

1:14:51way to output. my my my change my back propagation is never going to make it to these early layers >> because I've just been multiplying a number that's slightly smaller than one smaller than one I mean in effect I just use the idea of 0.9 to the 100 right sometimes it's.5 and then you're really screwed right >> so that's the vanishing gradient problem if you're like really bad at it >> okay we we solved this already but I'm just letting you know >> this is still kind of a problem >> you couldn't just say oh back propagating we're fine it's like well only if it's not greater than one. And also, >> so so here's the problem that happens if that if that number that you're multiplying is a little bit smaller than one. If it's larger than one, then you

1:15:31have the opposite problem, right? Because 1.1 to the 100 1.1 * 1.1 * 1.1 is something like 14,000. >> And so you have an exploding gradient, >> right? [clears throat] Where like one the system gets really chaotic. It's kind of like a chain reaction in like nuclear physics. Yes. Right. And there have been historical fixes like the long short-term memory LSTM model. Um it used something called a constant error carousol. But it's it's that thing was limited to sequential data. Okay. >> Right. >> Along came in 2015 ResNet. Okay. Residual network. >> Okay. This was a very big deal because they did a very simple fix. >> Okay.

1:16:11>> Okay. They said I'm not only going to have a feed forward layer. So it goes layer and then it does something goes to the next layer. I'm also just gonna pass along the input. >> Oh, okay.

ResNets and skip-connections as an “Euler method” view

1:16:22>> As in just the same thing. Yes. So, so the next the next part of the network, the next block is going to get my original input plus whatever computation I've done, >> right? Your your output from your part of the chain plus what you had as your input to get to that output. >> Yeah. Yeah. And that's that residual connection. That's that's why it's called the ResNet. Okay. I like to just call it the skip. >> The skip. >> Okay. The skip connection. All right. And the skip connection is very cool because this is basically doing something called Oilers's method, which is something that you learn in calculus. The idea is like if you've got a differential equation and you want to figure out where it's going, all you do is you say, "Okay, well the slope is this way. I'm going to make a step, but I'm going to keep going in that direction and then make a step and then

1:17:03keep going in that direction." Right? So that identity mapping, what it does is it ensures that you're not going to have that vanishing gradient problem. >> Right? Because even if it's like 0.99 to the D, you're still adding like it's still one plus or minus something, right? One plus or minus something. And you can still have an exploding gradient because you could still have that plus plus, but then you have stuff like batch norm and layer norm that can like keep that keep that in check, right? >> So this kind of solved that. Yes, the reset the ResNet problem kind of solved that. >> Okay, >> the problem with the ResNet is you want to do better. You always want to do better. Okay, in this case, I've got a

1:17:43one residual connection. It's like a single lane highway. All I'm doing is I'm taking the input, I do a bunch of computation, and then I add it to the input again. Right? This thing is a single lane highway. I'm not doing anything fancy here. >> Low bandwidth. >> Low bandwidth. I'm not doing anything fancy. 2024, 2025, a paper by bite dance, >> their AI, which is the Tik Tok company. >> Yes. >> Right. >> Yes. Yes. >> They come up with something called hyperconnections. >> Okay. which is what if we can start doing something fancy with that you know that skip connection instead of just like copy pasting >> right >> what if I do something fancy >> in that lane right okay and this is what this is what they did so on the left

1:18:24hand side is your original reset which is just copy paste copy paste do do some computation copy paste like that on the right hand side what they're doing is you can imagine instead of a single lane highway you've got multiple lanes and you're mixing traffic okay so you've all of this information and you want to find the optimal way to send this information to the next place. Right? This was the bite dance idea from hyperconnections. This was a paper in 2024 2025. Right? >> This is good. >> So the idea is to always have more and more computation in all of these little segregated spots because then that makes your model better, right? The more the more sort of >> fine-tuning you're doing here and there

1:19:05and here and there. Yeah. Yeah. You're getting better, right? the the model output is going to be better. >> First thing let it letting it run and then trying to like >> send back a conclusion that has now had these intermediate steps where there's no feedback loop. >> Yeah. There's no and and there's no it's it's kind of just lazy. Yeah. >> Right. It's just like copy paste. Well, but what if what if the next step could use some more finagling, right? What if I could finagle this this part to do a little bit better here, right? so that I get a better answer all the way at the end. Right. >> Right. That's that's the whole idea. >> Makes sense. >> Right. So this it's that multiple lanes kind of traffic merging and and the point is during training we figure out

1:19:46what the rules of the traffic merging are. Right. >> Mhm. >> Okay. The problem is kind of the same of what we had earlier which is

Hyperconnections (multi-stream “highway” for features)

1:19:57if you're the whole point of this skip connection was to make sure that I don't blow up and I don't >> vanish >> vanish. But now if I don't have that identity, I could start blowing up and vanishing here on this end. >> Right. It's the same problem that I had earlier. >> I gotcha. >> Okay. So, so it's it's hard to use these hyperconnections >> because the they were just being let loose a little bit like there like maybe there was a wide road but not lanes. >> Yeah. >> In between what like would be like lanes for different things to go down. I was just like, "Yeah, just everyone the wide road. Let's all go." >> And and if like and if the highway

1:20:37itself is now like magnifying the amount of traffic, >> then you're running into the same problem that you were earlier. >> That makes sense, >> right? Okay. And that is where this particular paper comes in. Okay. With something called a manifold solution. >> Okay. >> Okay. The idea is what you do is you say, "Okay, remember that hyperconnection which is a way to a way to change the how we're merging this traffic stuff, right? It's it's that thing is just

Constraining the mix to doubly-stochastic matrices

1:21:07another matrix. >> Okay. >> Okay. That thing is just another like a block of numbers that's going to tell me how much of this vector is going to go into that one, how much of this vector is going to go into that one." And the problem is that if this matrix amplifies or deamplifies then I get into that vanishing or exploding problem right so what if I constrain that matrix to basically never do that >> and that's what this is they they constrain the matrix to be doubly stochastic and what that means is the sum of all the rows is one the sum of all the columns is one the question is how do you how do you make a matrix do that it's not really a trivial uh way to do it but so it's It's it's kind of a

1:21:49brute force algorithm something called the um Synhorn Knop algorithm. All you do is you basically say okay take this row divide by the sum. So now everything sums to one. Next row divide by the sum okay and then you do it for the rows. Now you go along the columns then you go for the rows again. Then you do the columns and one by one. It's like you're folding a pastry over and over again until you get a very nice >> matrix where all of the rows and all the columns sum up to one. The key is once you do that, you're no longer going to blow up >> or shrink. >> And that's the key insight. That's it. >> That's it. It's just an iterative brute

1:22:29force method to constrain the hyperconnection. >> That's it. >> That was the idea. >> That was the idea is like, you know, we've got I I want to do this I want to do this hyperconnection stuff, but I don't want to deal with all of the nonsense of blowing up and all of that stuff. So I'm just going to put in a little hack where I make sure that these hyperconnections are constrained so that the sum is one. And so I'm not stretching, I'm not shrinking. So every time I'm not like doing 0.9 * 0.9, it's always 1 * 1 * 1. I'm just mixing. >> I'm mixing it in a in a really nice way. >> I'm I'm not mad. Sometimes simplicity, you know, >> it's a very simple hack. It it's one of

1:23:10those things where some people are probably looking at that paper being like, I can't believe you've done this. >> Yeah, it it's it seems pretty simple. Now um >> again within it's not obviously. >> Yeah. Yeah. >> But >> obviously, but like but um this this is very important for Deepseek though, right? Because Deepseek um >> Bingo. It's it's hard for them to get GPUs. There was this very recent news that actually came out a month ago where like two dudes got caught with $160 million.

Practical scaling realities: GPUs, export controls, “smuggling”

1:23:42This is uh Yeah. $160 million worth of GPUs that they were trying to that's >> smuggle into China. >> This was a month ago. >> So they're desperate, right? >> And and that's this is, you know, so for context, right? And I believe this started under the Biden administration. there was sort of this doctrine about how do we win the sort of great power war for AI. And the idea was because people really buy into the scaling laws as like the most accessible lever for us to pull and like the fastest way to sort of accelerate the progress. Uh we continue to double down on domestic chip production. We have the lead on that because of the likes of people like

1:24:23Nvidia who just have something no one else has. Um and we control all exports of these not only to countries like China but a whole list of others. And there was this really big conflict when Trump administration came in because the argument was being made that uh >> that that is not solving a problem. >> Yeah. >> Uh and then it's also limiting eliminating like the ability to generate revenue that can then be applied and put in other places. And so China, the the Chinese and the research community there are having to be clever with not enough because we haven't seen BYU or any of these larger, you know,

The memory wall: kernels, throughput, and systems tricks

1:25:04tech companies within China get to that point. They're getting they're trying to get there. >> They're trying to get there. >> They're trying to get there, but they have not yet. And that's still the bottleneck. >> Yeah. And I think I think very recently the Trump administration said that they're going to allow Nvidia to sell to China but they'd have to pay like a 25% tariff that would go to the government. So the government would like share in the profit. Something like that. I'm I'm not quite clear. But in any case, what we're trying to do is like choke out, >> right, >> the Chinese AI industry, right, by starving them of GPUs. So the Deep Sea guys have to be very clever. >> Yes. >> In how they implement this because

1:25:45here's the idea, right? Modern GPUs um like the Nvidia H100s, they're characterized by a massive disparity between how much they can compute. So the compute capability which is in number of flops, floatingoint operations and their memory bandwidth in like terabytes per second. Okay, this is called the memory wall which it's refers to the latency and the energy costs associated with like moving data from memory which is this high bandwidth RAM to like your spam registers which is the thing that >> h the computing part of the GPU has access to right and so it's really slow to like transfer stuff

1:26:26>> and if you're starved of GPUs you need to come up with clever ways to if you want to do something very cool like this manifold old hyperconnection thingy thingy thingy, but you're not you don't have a lot of GPUs. You got to get clever, right? And that part of the paper I actually thought was really cool, right? So, you've got this bottleneck, right? And >> if you want to naively do something like this hyperconnection stuff, >> you know, you're trying to before you just had a single lane, now you've got let's say four lanes. That's four times the compute that you have to do because you have to figure out you have to you have to read your your input then you have to churn the numbers four times

1:27:07>> right >> and then that's expanding a lot of a lot of >> comput four to four to five times higher than the original ResNet >> right >> okay and when you're starved with GPUs four to five times higher input output is not >> something that >> it's too inefficient >> yeah you can't just by >> right >> like 500 more right so what deepseat did was reduce that computational overhead to something that was negligible six to 6.7%. really >> instead of four to five times. And here's how they did it. They use this language called tile lang. >> Okay. >> Okay. And what they're doing is literally overcoming the memory wall by

1:27:49something called kernel fusion. So whenever you're doing AI, let's say you do like some kind of operation, you have to like start up a kernel. >> Yes. >> It's it's just like it's kind of like a processing unit. >> Yes. >> Okay. And individual kernels are started up for individual tasks. like there's matrix multiplication, there's the activation and stuff like that. But what these guys did was customize it so that all of the math is done in a single kernel faster arithmetic and the intermediate results don't go to memory back and forth. So you don't like you don't like do you don't like go to memory, get some data, do something, put it back in memory, go somewhere else, put it back. It's kind of like if you're

1:28:29like a chef, like an industrial chef, you don't go to the fridge all the time, right? It's like you get all your ingredients, you put it on your stove top. The fridge is like your high bandwidth memory. The stove top is like your SRAMM, which is the countertop. So that's it's it's limited, but you want to do everything you can in this in this little time. >> And the kernel is your instructions. But if you can make your kernel very complicated and clever, yes, >> then I don't have to go visit the fridge all the time, right? And that's exactly what they did. reduce the global memory reads by a huge amount. >> That's fascinating. Which then which then opens up the memory like the

1:29:09bandwidth on memory because you're not constantly going back to store and retrieve in time as you're doing any number of different operations or processes. >> Yeah. And the proof the proof is in the pudding. They've got a smoking gun. They use the 27 billion parameter model. The loss is lower. The gradients aren't like blowing up. Um, so it it is it is very

Why the quantum analogy shows up (unitarity = stability)

1:29:29cool, >> you know, but it's it's kind of it's kind of insane that like it was such a simple trick. It was just like, oh, just make the matrices like add up to one. >> Like just make sure it doesn't blow up. Like yeah, no, that was the whole point of ResNet and they just like did it again. [laughter] But it's not in the same way but like >> still you know the the the concept is very similar and it's it's very physics intuition as well because you know in quantum mechanics for this this reminded me a lot of quantum mechanics because um in quantum mechanics you have for example quantum states and then you have something called operators that operate on these quantum states like a time evolution operator would take your quantum state now and tell you what happens sometime later and sometime later and sometime later right Um,

1:30:12and these operators have a condition that they have to be unitary, meaning the probability has to add up to one. >> Okay. >> Okay. Cuz all of the outcomes need to be all of the outcomes. You can't have an operator that expands probability so that the sum of all probabilities is two. >> But there's a very central physical insight, right? It's like conservation of probability. You also have conservation of energy. In this case, they have like conservation of norm or something like that, you know? It's it's like a very simple insight. >> This is and and this goes back to the way you kind of always talk about and see the world which is like you can apply the the the >> the mental model that we have as physicists into other domains. >> Yeah. And it gives you it's like a way

1:30:54of thinking about stuff and systems. Yeah. And you know, order and progress and and it >> because it's funny that we again did the Princeton neuro story followed up by this one because there there it's like two different types >> types of manifold. Yeah. In this case, the manifold is like constrain it into this size. Yes. >> Right. There it was like a a a giant smooth sort of manifold. Here it's like just make it small. and and the what is and what was driving you know with the sort of deepse story it's it's obviously very much focused on like a commercial

1:31:34outcome uh the insight was >> not particularly uh not it didn't have a huge wow fac had wow factor not the the intuition >> yeah [clears throat] >> whereas I feel like with the the Princeton neuroscience story like the intuition >> intuition yeah was really really cool Um we I know every a lot of the AI community and the sort of tech community

Plain-English: what “manifold constrained” is buying you

1:31:58no one know none of them knew what man hyper uh whatever >> manifold constrained hypercon hyperconnections they everyone's like what is I don't know what manifolds are I don't know what constraint means in this context I don't know what hyper connections are >> but now now you do right hyperconnections is the mixing of the traffic the manifold constraint is just just say the thing that is mixing the traffic better not amplify the traffic or shrink the traffic. >> The traffic better be the same. >> This is obviously going to make models everyone's going to start in different ways capacities using this. It's going to make models much much better. >> Yeah. And and it's also like kind of a a shift in the paradigm of just like it's

1:32:40just compute compute, right? Because right here this is algorithmic innovation >> that's acting as a substitute for compute. You bring up a really good point because that's been the, you know, people sort of say, you know, the argument is like, oh, the Americans are just, you know, being fat and lazy because uh or being dumb and lazy in this context because they just want to throw money at the problem. >> Yeah. >> And then they're not actually doing any of the variety of areas of fundamental research. I mean, some people argue that you can't just scale with this is the Yan Lun, you know, other like you can't just get to >> whatever you want to define as AGI by scaling LLM. you need some or some subset even uh Dario at at Anthropic kind of same says the same thing. You need some subset of other insights or

1:33:21breakthroughs around it. >> Um [clears throat] but you know this is a perfect example of identifying that there's a lot of green space still >> to in these areas that people might not be focusing on because the rat race around being the first AGI is has people just like what is the easiest way to get there quickly throw money at the problem. >> Yeah. And so we don't get clever. Very interesting. Again, the USChina nexus is not going anywhere. >> No, >> it is what it is. Y >> um >> and so we we hit we've hit Rome. >> We went really far back in time. Then we went real real deep on the brain.

1:34:01>> Yeah. >> And then we kind of went a little bit we we went shallow seek on [laughter] >> Yeah. >> on AI, the fake brain. And now we're going to move to our last story of the day. Uh which we have aptly titled econo

Story 4 begins — econophysics and market impact

1:34:18physics. >> That's right. >> As you can see, we are dawned in our finance bro garb. >> Yeah. >> Um >> I got my Jackson Hole. [laughter] You know, bunch of finance bros there. Let me tell you, last time I went, >> and I have no idea where this is going to go. I was when I saw this I was um I was I was pleasantly surprised and it caught my eye because this was a paper in PRL physical review letters. This is the story journal that has published, you know, superc conductivity, the black hole papers, Einstein, Wheeler, um, cosmic microwave background, and then a universal law about stock prices

1:35:02and the impact that orders have on the price of an asset. And I was like, what is this doing in a physics journal? >> Mhm. >> Okay. >> Mhm. >> So, here's the central question. Okay. Um you know when when you trade stocks stons >> stons >> yeah when you trade stocks right um the the trade itself affects the price of the asset. >> Mhm. [clears throat] >> Okay. The question is how does that price change? Okay, there's something called the square root law which says that if if I if I put in an order for a certain amount, the amount that the

1:35:43price is going to change as a response to that order is proportional to the square root of the size of my order. >> Okay. And it's sort of just been observed, but this was a paper that was trying to figure out is it a square root? Meaning, is the exponent a 1/2 or is it 1/4 or is it like 04? Is it 6? What is it? Okay. And the the the field of econo physics is actually quite old. Okay. All the way back in 1900, there was this guy Luis Bashellier. He wrote he wrote his senior thesis his PhD thesis la speculation. [laughter] That's my French. >> I was gonna say

1:36:25>> but but it's it's a it's a physics thesis about speculation in the markets. >> He actually predated Brownian motion. Like this is 1900. So this is before Einstein's 1905 paper on Brownian motion. Um derived the diffusion equation. It's a precursor to Brownian motion. all of this stuff just to get rich off speculation, right? Um, >> we love it. >> And and that was sort of the birth of like trying to use statistical mechanics and insight from physics to try and talk about agents in a stock market in a in a market with, >> you know, people that are trying to buy and sell stuff. >> Yes. >> Okay. >> Um, in the 1990s, there was a guy,

1:37:06Eugene Stanley. He he actually figured out something called the inverse cubic law of price returns which means if you if you plot the returns that you get and the probability distribution of the returns based on like all of the assets everywhere, right? It's it goes down like returns to the -3 power. >> Okay. >> Okay. Meaning like if you if you want 10 times the returns, the probability of getting 10 times the returns is 1 divided by 10 the 3 compared to just getting the same return. >> Mhm. >> Does that make sense? >> Yes, it does. It does. >> Okay. So, and it's it's the same for like whether you wait one day, whether

1:37:48you wait 2 days, whether you wait 3 days. It's kind of weird. >> That is a little weird >> that like the probability distribution remains the same. >> Yeah, I don't like that. >> Okay. Um, another one was the probability density of transaction sizes. If you look at all of the transactions in the New York Stock Exchange, in the Paris exchange, in the London Stock Exchange, and you plot the trade size on the x-axis, and like how many trades happen at a certain size, at another certain size, then the exponent is -2.5.

Universality: why “same law across stocks” is shocking

1:38:22>> Okay, which means that no matter no matter what stock market you're looking at, >> it's the same trend. >> I don't like that. It's really weird, right? And whenever physicists look at something like this, it reminds them of something called universality. Okay? Universality means there's some weird underlying physics that's happening. >> Yes. >> That is invariant of the thing that is involved in the data collection. Here's what I mean. Like in universality, what we what we a lot of times think about is um phase transitions. So if you if you think about gases or or any substance that's going from liquid to to gas and

1:39:04it's it's at something called the critical point which is where the liquid and the gas phase are like sort of same same >> right

The square-root law defined (impact ∝ √volume)

1:39:12>> and you plot you plot something like the um the specific heat of this substance. Okay. As as you change some parameter like temperature or density or something like that. Regardless of whether you're using helium or oxygen or this or that, the shape of that curve is going to be exactly the same. Which means there's some underlying physics that is invariant of >> the the actual mechanics of the particles >> itself. >> Yeah. There's like there's something much deeper. Okay. So whenever we see stuff like that, we're like there's something. Okay. And and that's I think why this made it to PRL. >> That makes sense. >> Okay. That makes sense. I think that's why this made it to the physical review.

1:39:54Okay, so the um the empirical law is that square root law. Okay, it's that if I if I put in a buy order of some size, let's say X, and then I put in and then somebody else puts in a buy order that's four times, right? My buy order is going to move the price some amount,

The intuition: why doubling size doesn’t double impact

1:40:20but their buy order is not going to move the price by four times. It's only going to move the price by twice what mine moved. >> Okay? >> Because they did four and the square root of four is two. If somebody put in nine times my order, it would only move the price by three. Mhm. >> And it kind of makes sense >> because, you know, if if some asset is like trading at $10 and then I put I put in I put in like uh I want to buy something at like 10.1, right? There's not going to be a lot of people who are willing to sell it to me. >> Mhm. >> So, the price is really going to go to 10.1, right? But if I if I put in an order for like I want to buy like a

1:41:00bunch a bunch at like 12 or something, right? There's going to be a lot of sellers that are going to be really keen and so the price is not going to move that much because a bunch of sellers just like fill that order. Yes. Right. So that's the the intuition behind it. >> The idea is that it's nonlinear in that sense. Right. The the more that I the more volume that I want to buy or sell, the less the price is going to get impacted. It's not linear. linear would mean that it's directly proportional. Here it's like a diminishing return. Yes. >> Type thing. >> Yes. >> Okay. So the question is is this truly universal right? >> Yes. >> Now in order to actually um measure something like this, right? I mean there's a lot of people who think that

1:41:41it's non not totally universal, right? Maybe it depends on the micro structure like there's inventory risk models like how much inventory is out there in the market. Um, like if there's large traders, like what if there's like a giant whale, you know, when we talk about Bitcoin whales, like like if there's a if there's a market where there's only big orders, >> right? No one's buying like small stuff and it doesn't even matter if you're buying small stuff, then does that change the dynamics of the square root law right? >> Okay. The univers which makes which makes >> which totally makes sense. But the universality hypothesis says that no, it's still 0.5, right? the the exponent is always going to be square root. >> Okay? So,

1:42:21>> um to discern between whether it's one or the other, you have to get some kind of very good data. Okay? Because normally when when we want to analyze data, there's these things called meta orders, right? Like there's a hidden parent order and then there's anonymous like child orders. Like if I want to buy like 10 10 stocks of a particular asset,

Metaorders + reconstructing true trader intent

1:42:43I could just put in an order for 10. Or I could be like, "Okay, I want to buy two and then I'm going to buy three and then I'm going to buy two and then I'm going to buy three." That's four different little child orders for one big meta order, >> right? Which is which is that you bought 10. >> Yeah. Yeah. And um in normal in normal sort of trading in the data that you get if you want to analyze this kind of stuff, you only have access to the little orders. you don't have access to the guy who's right because each of these each of these um each of these people have like a virtual server. So each of the each of the little orders is might have like a virtual server ID and so it's going to be like pretty impossible to reconstruct who did >> which one. Right. >> Right. >> Okay. So this is where the Sato Kanazawa

1:43:25study >> that was just published in PRL um in 2025 in December 2025 comes in strict universality. They show that this is a very strict universality and it's a complete survey of the Tokyo Stock Exchange. Okay, so the Tokyo Stock Exchange what they did was they got a bunch of data from that stock exchange from 8 years. So from 2012 all the way to 2019 and the key innovation is they figured out a way to stitch together the effective trader IDs, so the big meta orders from these virtual server IDs. >> They kind of deanonymized it basically. >> Yes. Yeah, it's still anonymous, but they can pull them together

1:44:07>> to recognize a single trader. And the way they did it, the reconstruction method is like actually pretty pretty cool. Okay, so here's what they did. They developed a heristic that said, okay, I'm going to aggregate ID such that if server A submits an order and then server B cancels that order, the only way that's possible is if both of these servers are connected to the same trader. So every other time [laughter]

Results: exponent ~1/2 across stocks and traders

1:44:33I'm going to treat that. Yeah. >> Right. As it's pretty good. >> That's really good. >> And so you get these effective trader [laughter] IDs. You get you get the trader desk. >> Yes. >> Right. In these like firms. Yes. >> Right. And this allows them to to analyze the behavior of all the active traders across all liquid stocks, all 2,000 stocks for nearly a decade. It's like a god view >> of the market. >> I love that. >> Of the Tokyo Stock Exchange, right? Yeah. Yeah. >> Very cool. >> That's and that's clever. And like it again it it doesn't take a large leap to get Yeah. that connection. >> Yeah. Yeah. Yeah. But they did the due diligence, right? And they actually did it. >> And so when they when they actually go

1:45:13through and do the math, >> they find that the mean exponent >> is about 0.489, very close to 0.5. >> The standard deviation is 0.071. And there on the left you can see all of the different stocks they use. Toyota, um, NT, I don't know that that stock, but it's all of them are >> the same line. >> Yes. >> And they're all centered at 0.5. And if you look across all 20,000 stocks, the spread is quite big. I'll I'll admit it's like from 0.4 all the way to 0.6. But the mean of that spread is very sharply at 0.5. >> Okay. >> Mhm. >> So from this data set, they can actually

1:45:53falsify some of the earlier models. There's this one model called the GG PS model which is the I don't know go bikes go Krishnan Pluro Stanley model from 2003. This is the model of the inventory risk. This is the one that's saying that like okay market makers they provide liquidity but they face the risk right because like if I if I'm a market maker mean meaning like I have like I'm the one if if you're trying to sell I'll buy from you and if you're trying to buy I'm going to >> I'll sell to you and I'm going to make money on the spread. But if the price like >> goes off, then I'm just left with >> the bag. >> The bag, right? And I that I didn't want or maybe I did want. Okay. So, they

1:46:34charge a premium and that price impact is going to compensate for that risk. >> Okay. So, what if what if this stuff and the scaling argument is the size of the trades follows a power law, but to limit their risk, >> they're going to adjust the price to balance the probability of large orders. >> Mhm. >> Okay. Mhm. >> And there there's another sort of numerical coefficient that comes in beta that you can figure out, which is sort of the the probability of these large orders. And they say that the alpha, which is the the exponent of my price impact, the 1/2 that we're talking about here, the square root, is related to this beta. >> So these guys plotted those two, and there's no correlation.

1:47:16>> It's flat. >> It's flat. [laughter] >> Okay. So this 2003 model, this GGPS model, completely wrong. >> Nonsense. Yeah. Okay. Like most things in economics, but okay. [laughter] >> Um, >> you know, >> there's something wrong with the mic. >> So, zero correlation there. There was also another um fair pricing efficiency model. >> Zero. That's crazy. >> Yeah. Yeah. Zero correlation, right? Yeah. So, there's this FGLW model, which is the farmer Garig Lelo Wan Broic model that also was falsified. I'm not going to get into that, but it's about market efficiency and fair pricing.

Latent order books + reaction–diffusion model

1:47:53So I I was reading this and I was like, okay, I I would like to know mechanistically though what's going on. >> Right. >> Okay. I'm a physicist. It's like what's actually going on, >> right? >> Right. >> Because there's clearly some underlying >> Yes. The underlying physics that's there. >> There's there's some explanation. >> That's none of these other things. >> Yes. Okay. So the victor comes from something called latent order book models. Okay. This is from Bousard Doniier and Mastrom Mateo in 2015. Buchard actually holds a PhD in theoretical physics from ENS which is like the um you know top university in Paris and his analogy is um reaction diffusion in chemical reactions. Okay,

1:48:33imagine you've got two different types of particles. You've got a biparticle and you've got a cell particle. If they meet up they annihilate, right? So they they they they poof out of existence, right? The by particles are on one side and the cell particles are on the other side. Okay? And they all want to get to each other, but because of the nature of the reaction in the middle, there's not going to be any because if they were in the middle, they would annihilate, right? So, you're going to get a density profile that is roughly linear >> on the right hand side and the left hand side. And I think the next picture shows this very well, which is the density profile being comp like very much linear. So you're asking for a certain

1:49:15price above the current price and that's the that's in the red. So this is like this is like how much I would sell for. >> Yes. >> And then in the blue is how much I would buy for. >> Mhm. >> And it's roughly linear. >> Yes. >> Right. Okay. So now imagine with this linear sort of density profile of my um order book. These are the order books. Right. Suppose some order of size Q comes in. Okay. How much of it is going to go away? It's going to be that triangle. >> Mhm. [clears throat] >> Right. >> Mhm. >> And what's the area of the triangle? The area of the triangle is 1/2 times the

1:49:58length and the width. Right. So it's it's the total size is my area and that's proportional to the price impact squared. I'm so >> so the price impact is proportional to the area square root. >> Yep. >> It's it's a >> it's strictly universal. >> It's like it it like makes sense. It has to do with the shape of the >> Yes. >> of the or of the order book, >> right? >> Right. The area is the price impact squared. >> Yes. >> And so the price impact is the area square root. And the area is the size of your order. >> So can we all go home? >> It's [laughter] like pretty simple. I thought that was like kind of cute.

1:50:39>> No, that that is that is because it it it you know there's all a lot of always a lot of handwaving. Yeah. The thing I love about F it's like no but like like >> Yeah. How how does it work though? >> Show me like don't well like we have to you know some risk of >> Yeah. No, it's like no just oh it's okay. It's [laughter] just an area. >> That's why you guys don't want to talk about it because you're trying to take your premiums because then because you know anyway. >> Yeah. >> Um >> that was this is actually really cool. econo physics. >> Yeah. Yeah. It I thought I thought, you know, and it makes sense then now that it was it was in PRL. It's a pretty cool study like the way that they batched the orders together to like figure out what the trading desk was doing, things like that. I I think I think it was a really

1:51:20cool study. um you know the practical impact I don't know somebody's probably going to make billions in slippage yeah based on this or it's probably going to be priced in because now this is now that this it's it probably already is priced in because like the economists don't actually care about like the the finance guys don't actually care about the underlying like oh why is it doing it's like oh it's already been an empirical observation so they're going to put it in their algorithms anyway so you know I I don't know if like >> you're going to make a lot of money off this insight but it's a cool insight nonetheless This is not financial advice. >> No. [laughter] Yeah. Yeah. Don't go do experiments. Hey, I wonder if I buy a 100 Bitcoin, what's going to happen? [laughter] >> We're just some physics nerds over here.

1:52:01Okay. So, uh, >> don't come cry. No, that's actually that's because I think you can find there's structure all around us. Yeah. Right. Right. >> Exactly. Yes. There's structure all around us. And it's it's it's accessible, but it is not it's sometimes maybe a little opaque >> or or or or discreet in terms of how we engage in a like move around in the world >> and then how it becomes obvious. >> That's a very good point. There is structure all around us. I like that. >> It's it's and I mean this is through all

Big implications for markets, models, and “finance bro-ology”

1:52:31of our stories. We started with concrete. >> Mhm. >> Right. And that's pretty good with structure. Then we went to cognitive Legos and the structure of manifolds in our brain in our brain and biological beings. Then we went to the similar kind of structure for manifolds in constrained hyperconnections >> in the the human brain we're trying to replicate in silica. >> Mhm. >> And then the structure of this whole thing people view as like you need to be galaxyrained to beat the markets and all of this. It's like it's actually just >> it's just a area [laughter] >> of the triangle >> of a triangle. >> We hope you guys are excited this season

1:53:112. We have a lot cooking. As you can tell, we've tried to do some updates and upgrades. Not a perfect execution, but we've learned a lot in this first episode. We are trying to do longer overlays now with better viewability because we do know many of you want to actually digest the graphics and see it. For our listeners who are listening on the pod, we do we are a video podcast. We're available on Spotify. We're available on YouTube for our fulllength episodes and clips all over the socials. If you really want to kind of gro what we're talking through, especially some of the stories in today's episode, the graphics are super super helpful. Christian, you always do a good job of trying to verbalize what's being seen, but there's so much detail that's lost

1:53:52in it. >> And I did mean to bring this up at the beginning of the episode, but there were too many new variables that I was worried about that I forgot to. However, uh if you made it to the end, you get to know this before anybody else. Our new website has just gone live, and there's a sort of a couple of interesting things. So obviously if you want to support the pod you can go there and do so but all of our episodes are there and we are backlogging all of the research papers for season 1 and on an ongoing basis we'll be putting all of the research papers with key takeaways the related episodes all in one place so you can listen to us while you're on your commute to work or while you're in the middle of the gym workout and if you're like I actually am really kind of that

1:54:33kind of really picked my brain I want to dig in more you have an easy access to the source the sort of nature paper that it's from or the journal, the publisher, impact factor. We have some really really cool ideas for how we can >> start to do some interesting things with the leaderboard and rankings. I won't leak too much. Yeah. >> Uh we also we also did get confirmation for our first >> uh on-site >> I'll zip the lip. >> Yeah. Yeah. Yeah. That's it. >> It's all you get. But you can only get it if you go to the end of the podcast. Or maybe we'll sprinkle it in the middle. Who knows? So, you got to listen cuz you never know what you're going to get. Uh, this was a good long 2-hour

1:55:14epic. >> Oh, wait. We need we need to tell the listeners if they made it this far what to comment. >> Ah, yes. Yes. Okay, this is good. I'll if you have an idea, I'll let you I'll let you cook on it. I don't know if I um I >> Yeah, if you guys write down um finance and physics. No, that is No, no, don't don't do that. >> Don't do that. That's so bad. We're gonna edit that out. That was [laughter] >> This is why I don't do that. So, I don't know why you gave me the You know like >> you're the promo guy. Come on, dude. Do what you um uh I I my brain You fried my

1:55:55brain so hard today. >> Yeah, we've been working like really hard trying to get all of this set up for this season. So, >> and so so what may maybe let's do this. Um uh there there's structure all around us. >> Ah and a pun on that or something like

Wrap-up — what we learned this week

1:56:10that cuz last time there were some comedians. >> Yo yo some of y'all were hilarious. >> Yeah. >> Yeah. Well, you still have to put a compilation for that feedback. But >> yeah, there's structure all around us or something like that >> or something along those lines. The last thing I'll say is um we are hoping to in short order put a submissions page on that new website that I mentioned. So if you uh are published if you have a peerreview researched paper in a journal that has an impact factor that is greater than we will tell you soon but it's not going to be it's going to be pretty high just cuz we have to kind of keep things please send it to our submission page or if there's just something you're interested send it there too. If it is not in a journal uh

1:56:51that has a significantly high impact factor we will not cover it. It is not personal. It may be amazing and if it is amazing, it will ultimately end up in one of those journals and we will ultimately cover it. Uh we are just the two of us. We're trying to do what we can but we know so many of you send us stuff. We're trying to start the process of creating an ecosystem and community to begin to do so. So >> a lot of fun exciting things. Uh go ahead. >> Yeah. And oh, I was just going to say like the the one thing that I was going to say with the impact factor stuff is like we we cover um papers from even like lower impact factor >> to me peerreview journals. Okay. It's got to be it's got to be it's got to be

1:57:33peer-reviewed. >> We need reviewer number two. >> Yeah. Yeah. Archive archive doesn't work guys. It's it's not not today. >> Yeah. [laughter] >> Not. But hey, if you support the pod and help us grow, maybe one day we'll expand. We'll have a soon we'll have a big board behind us uh where we could get the infographics from the infographic team and we'll have the 3D animations like MKBHD just where we'll be like scaled down the nano scale. >> Um also we have a blank wall behind us

Outro + closing notes

1:58:00now because of the new angle. If you have anything you would like to put on the wall, please DM us. M as always I am your host Lester Nar joined by the one the only our resident PhD Krishna Chowdery we really appreciate you guys joining us and coming back and if you're a new fan who heard up from us or about us on Rogan uh you're in for a good treat. This is from first [music] principles. [music]

1:58:38[music]