Constraining the mix to doubly-stochastic matrices

1:21:07another matrix. >> Okay. >> Okay. That thing is just another like a block of numbers that's going to tell me how much of this vector is going to go into that one, how much of this vector is going to go into that one." And the problem is that if this matrix amplifies or deamplifies then I get into that vanishing or exploding problem right so what if I constrain that matrix to basically never do that >> and that's what this is they they constrain the matrix to be doubly stochastic and what that means is the sum of all the rows is one the sum of all the columns is one the question is how do you how do you make a matrix do that it's not really a trivial uh way to do it but so it's It's it's kind of a

1:21:49brute force algorithm something called the um Synhorn Knop algorithm. All you do is you basically say okay take this row divide by the sum. So now everything sums to one. Next row divide by the sum okay and then you do it for the rows. Now you go along the columns then you go for the rows again. Then you do the columns and one by one. It's like you're folding a pastry over and over again until you get a very nice >> matrix where all of the rows and all the columns sum up to one. The key is once you do that, you're no longer going to blow up >> or shrink. >> And that's the key insight. That's it. >> That's it. It's just an iterative brute

1:22:29force method to constrain the hyperconnection. >> That's it. >> That was the idea. >> That was the idea is like, you know, we've got I I want to do this I want to do this hyperconnection stuff, but I don't want to deal with all of the nonsense of blowing up and all of that stuff. So I'm just going to put in a little hack where I make sure that these hyperconnections are constrained so that the sum is one. And so I'm not stretching, I'm not shrinking. So every time I'm not like doing 0.9 * 0.9, it's always 1 * 1 * 1. I'm just mixing. >> I'm mixing it in a in a really nice way. >> I'm I'm not mad. Sometimes simplicity, you know, >> it's a very simple hack. It it's one of

1:23:10those things where some people are probably looking at that paper being like, I can't believe you've done this. >> Yeah, it it's it seems pretty simple. Now um >> again within it's not obviously. >> Yeah. Yeah. >> But >> obviously, but like but um this this is very important for Deepseek though, right? Because Deepseek um >> Bingo. It's it's hard for them to get GPUs. There was this very recent news that actually came out a month ago where like two dudes got caught with $160 million.