Compression index & measuring “compositional efficiency”

1:03:17>> It's cool. They they had also this thing called a compression index. What they found was like whatever the task is not not yet sorry but like whatever the task is relevant for like suppose it was paying attention to color. Yes. then the shape manifold would actually like shrink. >> Okay. >> Okay. Like if you look at the neurons that were doing the shape stuff, they wouldn't they wouldn't be as active anymore >> because because the the the the subject is paying attention to the color first >> and then and then so so even the the the manifolds themselves are dynamic in that sense, right? The neurons that are in this plane are now a little bit shut off because they they're not as important right now >> anymore or right right now, >> you know, in in this particular task. >> Yes. I mean it's kind of like muscles,

1:03:59right? If you're not like you're not using them, they kind of get a little smaller, but but they're still kind of >> Yeah. But this is Yeah. And but this is like, you know, on the order of minutes that they're like responding because because the subject knows which task is happening >> happening in real time and then subsequently which one to then prioritize or have active in a more active state. >> Yeah. >> Very cool. what >> it's very cool to see just how how our brain has like >> yes >> done this you know millions of years of evolution and um we're getting very close to solving like stuff like the binding problem which is a very big deal you know >> yes >> and this has a lot of implications for

1:04:40artificial intelligence >> I mean because because catastrophic forgetting is very much a thing right so um there was actually this there's a new way of trying to figure out >> how to um not do catastrophic forgetting. So if we go to photo 15, right, there's something called orthogonal subspace learning, which is the idea that if I want to train on one task, task A, and then I also want to train on task B, what I can do is I can make the two subspaces orthogonal inside my neural network. And now when I do back propagation, which we're going to get into later, but it's just basically a way to train the model and change the weights of the model, the the way that

1:05:24the model is going to train for task A and the way that the model is going to train for task B are going to be orthogonal. So they're not going to interfere with each other. >> And you can enforce that constraint. >> I get >> like I'm I'm if I'm on task A, I'm only going to move along this plane >> in my parameter space. And if [snorts] I'm trying to do task B, I'm only going to move along this plane in my parameter space, right? And that way I don't forget how I did task A, right? If I now all of a sudden start trying to learn task B, >> it's it's putting in uh roads and traffic lights into the pathways of uh h how >> yeah, >> the processes proceed in a way that they don't interfere with each other.

1:06:05>> This is so cool. >> So yeah, it's um it's it's really cool. I mean, there's this new there's this new AI architecture called orthogonal low rank adaptation. O Lara or Oura it very recently came out. I think it was last year. >> Mhm. >> Where they're trying to do do this >> because the orthogonal the the orthogonal idea this like orthogonal subspace learning conceptually that maps to the same thing you were just talking about earlier about when you have plane this plane and the like that's the orthogonal that's what it it's literally and they're applying it now in the context of training >> but it >> we are we're able to experimentally see this >> see this in that's that is >> and like straight up confirm it.

1:06:46>> Right. Right. Yeah, >> that is very good. >> I thought it I thought it was a very cool play. >> I mean, it's hard to beat Princeton. Yeah. At anything. I mean, maybe maybe football. You might get us in football >> like all the time, >> but um when it comes to being the leaders in a variety of hard sciences. My My brain is currently having a lot of orthogonal subspace. >> Yeah. Yeah. Every time we get on this podcast, I have to do that. a lot of orthogonal subspace learning [laughter] but it's it's very cool and I think um you know in terms of so this is not just for um fundamental science right you might be asking okay like

1:07:26>> cool guys >> cool guys right [laughter] but you know cognitive inflexibility is um one of the symptoms of psychiatric disorders like schizophrenia and ASD right and you know >> given this kind of fundamental research it could be that that inflexibility comes from the failure here to orthogonally separate and compress [clears throat] subspaces in your head. So you can imagine like um non-invasive neurom modulation therapies that would like orthogonalize subspaces like make it more perpendicular. >> You have these headbands on that would yeah that would like that do the thing but but the point is there's a big

1:08:07>> I mean and that's that's far out but like you got you got to get to this kind of fundamental research. >> You have to understand that the manifold is there. [laughter] >> Yeah.