The key concern — is reinforcement learning doing the real work?

1:00:13>> Here's the concern. There are eight input channels for Pong, right? But Pong is a very small game. >> I can reasonably put all of that information into eight electrodes. How do I encode everything that's going on in Doom with only eight inputs? >> Yeah. each of which can only have a few possible values like either the electrode is on or off. And the answer is you actually don't. >> You use a reinforcement learning model. >> This is where it gets a little bit fishy. >> Okay. >> Okay. >> Here this is from their this is from their doom neuron. Um like they've got a little >> it's kind of like a GitHub where they have a documentation on what they did. This is the architecture of what they're

1:00:53showing is they've got a computation unit and that is talking to their CL1 computer. >> Yeah. Yeah. >> Okay. In that computational unit on the left right hand side, what you can see is there is a PO policy >> which is a reinforcement learning paradigm >> that is then encoding >> the input of the game that is then going as stimulation to the brain to the to my brain in a petri dish and what I get back is action and that is getting fed back into my reinforcement learning algorithm. So now I can very easily ask who is actually doing the learning >> correct? Is it my reinforcement learning network or is it the actual biological neurons that I have? >> Doesn't count, >> right? It shouldn't really count. And

1:01:34it's not it's not that far away. Like reinforcement learning, there's video games that have like perfected reinforcement learning. Dota 2, OpenAI defeats Dota 2 World Champions. So that's not that big of a deal. Correct. >> So >> the question is, was the software decoder, which is that translation layer, >> Yeah. Was that actually in the the learning algorithm or is the actual neuron doing something? >> Cuz in the pawn case that we already talked about in the earlier direct it was direct and it was maybe incremental but it was demonstrable learning. >> Yes. And and we could very visibly say the neurons are doing the computation. >> The neurons are doing the learning. >> The neurons are doing the learning because there's no funny business going

1:02:16on where like I've got a reinforcement learning agent that okay fine. So now it's playing a video game. What's the big deal? knows all the history of Doom's lore internally. >> Exly. So, here's how they address that. They actually did address it in their um >> frequently asked questions section. They said, "Okay, isn't the decoder doing all of the learning?" What they tested it was the how they tested it was through an ablation study. So, they've got a metric called a breakthrough rate, which is the amount of episodes that scores above a negative 600 reward threshold. Now,600 is pretty low because 1,000 is the floor. I've never played Dune Doom. So maybe somebody can tell me ifgative600 is like a lot or a little.

1:02:57But here's the key metric that they said. They said that if I were to use the neurons in the loop, I get 27% breakthrough rate. So my score is above that 600. If it's complete silence, so no input from the neurons. Or if I get put random noise from the neurons, then