Manifold-constrained hyperconnections (mHC) overview

1:10:44is what deepseek is purporting to have found. Okay, it's something called manifold constrained hyperconnections. It's a new way to train models more efficiently and actually like straight up better. Okay, but it's a very small hack that they're doing and it's actually quite simple. >> The hack that they're the the little hack that they did is like really simple but >> apparently it's a very big deal. >> Okay. >> Okay. So, let's get into it. >> Okay. >> Neural networks. First, let's just talk about neural networks real quick. Okay. Neural networks are a giant mathematical function that basically it's fancy linear algebra. Okay, this is um a

1:11:27transformer architecture from the very famous paper attention is all you need, >> right? >> Sorry, I repeated after him. I almost got through the whole episode without doing it. >> It's hard to unlearn 15 years of friendship. Please. >> Yeah. Yeah. Yeah. Well, so all of this is showing all of these arrows are basically numbers going from one thing to the other and then each block is effectively a matrix. You're multiplying matrices, then you add stuff, then you multiply a matrix, then you do a threshold, then you add stuff. It's a giant mathematical function at the end of the day. And each of the building blocks of these are something called an artificial neuron. Okay? And an artificial neuron is effectively kind of

1:12:10like a biological neuron in that it sums up its inputs and then if that input is above a certain threshold, well depending on, you know, the activation function, but let's just say it's above a certain threshold, then it lets it through. If it's below a certain threshold, then it doesn't let anything through. So that's that nonlinearity part. Okay? [clears throat] >> Mhm. >> And when we train a network, we've got these billions of parameters, billions of neurons that are all connected together. And what we want to do is something called back propagation. So you've got a network with weights which are how these neurons are connected. Um