Vanishing gradients and Jacobian products

1:14:10>> Now, if the numbers that you're multiplying are a bit less than one, >> let's say 0.9. >> Mhm. >> But there's 100 layers, 0.9 * 0.9* 0.9 100 times is something like 0.00003. This is the vanishing gradient problem. Interesting. Okay. The idea is the the stuff that's changing, the weights that are going to change if I do this very naively, >> yes, >> is only going to be towards the output, >> okay, >> of my of my neural network. All of the earlier layers that I have like, you know, it goes input and then there's something something something all the

1:14:51way to output. my my my change my back propagation is never going to make it to these early layers >> because I've just been multiplying a number that's slightly smaller than one smaller than one I mean in effect I just use the idea of 0.9 to the 100 right sometimes it's.5 and then you're really screwed right >> so that's the vanishing gradient problem if you're like really bad at it >> okay we we solved this already but I'm just letting you know >> this is still kind of a problem >> you couldn't just say oh back propagating we're fine it's like well only if it's not greater than one. And also, >> so so here's the problem that happens if that if that number that you're multiplying is a little bit smaller than one. If it's larger than one, then you

1:15:31have the opposite problem, right? Because 1.1 to the 100 1.1 * 1.1 * 1.1 is something like 14,000. >> And so you have an exploding gradient, >> right? [clears throat] Where like one the system gets really chaotic. It's kind of like a chain reaction in like nuclear physics. Yes. Right. And there have been historical fixes like the long short-term memory LSTM model. Um it used something called a constant error carousol. But it's it's that thing was limited to sequential data. Okay. >> Right. >> Along came in 2015 ResNet. Okay. Residual network. >> Okay. This was a very big deal because they did a very simple fix. >> Okay.

1:16:11>> Okay. They said I'm not only going to have a feed forward layer. So it goes layer and then it does something goes to the next layer. I'm also just gonna pass along the input. >> Oh, okay.