ResNets and skip-connections as an “Euler method” view

1:16:22>> As in just the same thing. Yes. So, so the next the next part of the network, the next block is going to get my original input plus whatever computation I've done, >> right? Your your output from your part of the chain plus what you had as your input to get to that output. >> Yeah. Yeah. And that's that residual connection. That's that's why it's called the ResNet. Okay. I like to just call it the skip. >> The skip. >> Okay. The skip connection. All right. And the skip connection is very cool because this is basically doing something called Oilers's method, which is something that you learn in calculus. The idea is like if you've got a differential equation and you want to figure out where it's going, all you do is you say, "Okay, well the slope is this way. I'm going to make a step, but I'm going to keep going in that direction and then make a step and then

1:17:03keep going in that direction." Right? So that identity mapping, what it does is it ensures that you're not going to have that vanishing gradient problem. >> Right? Because even if it's like 0.99 to the D, you're still adding like it's still one plus or minus something, right? One plus or minus something. And you can still have an exploding gradient because you could still have that plus plus, but then you have stuff like batch norm and layer norm that can like keep that keep that in check, right? >> So this kind of solved that. Yes, the reset the ResNet problem kind of solved that. >> Okay, >> the problem with the ResNet is you want to do better. You always want to do better. Okay, in this case, I've got a

1:17:43one residual connection. It's like a single lane highway. All I'm doing is I'm taking the input, I do a bunch of computation, and then I add it to the input again. Right? This thing is a single lane highway. I'm not doing anything fancy here. >> Low bandwidth. >> Low bandwidth. I'm not doing anything fancy. 2024, 2025, a paper by bite dance, >> their AI, which is the Tik Tok company. >> Yes. >> Right. >> Yes. Yes. >> They come up with something called hyperconnections. >> Okay. which is what if we can start doing something fancy with that you know that skip connection instead of just like copy pasting >> right >> what if I do something fancy >> in that lane right okay and this is what this is what they did so on the left

1:18:24hand side is your original reset which is just copy paste copy paste do do some computation copy paste like that on the right hand side what they're doing is you can imagine instead of a single lane highway you've got multiple lanes and you're mixing traffic okay so you've all of this information and you want to find the optimal way to send this information to the next place. Right? This was the bite dance idea from hyperconnections. This was a paper in 2024 2025. Right? >> This is good. >> So the idea is to always have more and more computation in all of these little segregated spots because then that makes your model better, right? The more the more sort of >> fine-tuning you're doing here and there

1:19:05and here and there. Yeah. Yeah. You're getting better, right? the the model output is going to be better. >> First thing let it letting it run and then trying to like >> send back a conclusion that has now had these intermediate steps where there's no feedback loop. >> Yeah. There's no and and there's no it's it's kind of just lazy. Yeah. >> Right. It's just like copy paste. Well, but what if what if the next step could use some more finagling, right? What if I could finagle this this part to do a little bit better here, right? so that I get a better answer all the way at the end. Right. >> Right. That's that's the whole idea. >> Makes sense. >> Right. So this it's that multiple lanes kind of traffic merging and and the point is during training we figure out

1:19:46what the rules of the traffic merging are. Right. >> Mhm. >> Okay. The problem is kind of the same of what we had earlier which is