Print Initials Understanding with airSlate SignNow

Print initials understanding

this is the python code needed to create and train a neural network that detects handwritten digits with over 90 accuracy no worries i'll explain and animate everything in detail so you'll understand the code by the end of this video a neural network consists of a bunch of neurons that are connected through weights one column of neurons is also called a layer there are three different types the first type is called input layer and is used to represent the input values that are passed on to the neural network the second type is called hidden layer a hidden layer receives values from a preceding layer does some magic and passes its neuron values to the subsequent layer a neural network can have zero to well basically unlimited hidden layers the third type is called output layer and is used to represent the output values it is basically the same as a hidden layer except that its neuron values are used as output and are not passed onto a subsequent layer when each neuron in a layer is connected with each neuron in the next layer as shown here the layers are fully connected in this example the input layer is fully connected with the hidden layer and the hidden layer is fully connected with the output layer there are other ways to connect two layers but fully connected layers are most used okay and what exactly do neurons represent just numbers in our case the input neurons are represented by pixel values of an image and the hidden and output neurons are calculated using the input values and weights but more about that later for now just remember that a neuron is just a value what are the weights then well as the name suggests just numbers that are randomly initialized a good practice to do this is using a range of small random values with a median close to zero for example minus 0.5 to 0.5 to initialize the weights we also need to know the number of neurons in each layer which are 5 4 and 3. therefore the weight matrix connecting the input layer with the hidden layer has a shape of 4x5 while the weight matrix connecting the hidden layer with the output layer has a shape of 3 by 4. defining the matrix from the right layer to the left layer is not quite intuitive at first but it is the recommended way and results in cleaner and faster computations later so i'd recommend sticking to this order there's one more thing to do when creating a new network let me introduce the bias neuron this is a special neuron that is always 1 and only has outgoing weights remember that the other neurons also have incoming weights initializing the bias weights is like initializing the weights for the other neurons with the difference that the initial values for the weights are zero this is because we want to start with an unbiased neural network but why exactly do we need the bias the idea behind it can be shown when looking at a graph think about a very simple neural network with only one neuron and no hidden layer this network can only learn a linear function while the normal weight determines the slope of the function the bias weight allows the function to shift up and down this means that the neural network is only able to accurately distinguish the circles from the crosses when it makes use of the bias neuron now that we know how a neural network is structured we can take a closer look at the images used for training the images itself are of size 28 by 28 which means they consist of 784 values this means that our new network is way too small so for the animations we'll just use 5 instead of all 784 values to train it on all values we basically just need to increase the number of weights in the code to later train the neural network each image needs a so-called label that specifies which number the image represents in this example the label is zero you're probably wondering where the images and labels are from and how many images we have for training well classifying images has been a challenging problem a few decades ago to figure out how good an algorithm is compared to other algorithms some researchers collected 60 000 handwritten images converted them into 28x28 grayscale images and paid some fellow humans to label them they gave it the name mnist database which stands for modified national institute of standards and technology database and published it online where it became widely popular today it can be thought of as the hello world dataset for machine learning to get the images and labels into our python program we need to execute this line of code it fills the first variable with 60 000 images each of which consisting of 784 values therefore it has a shape of sixty thousand by seven hundred eighty four the second variable is filled with the labels which we expect to be of size sixty thousand by one but if we were to execute this line we'd see that the shape is sixty thousand by ten this is because as soon as we have a classification problem with more than two possible outputs we need to represent the labels in a binary format also called one hot encoded to illustrate this let's assume we want to classify an image which has the label 3. since we have 10 possible labels in general we need 10 output neurons in our neural network and if our neural network is trained perfectly we'd expect all output neurons to be zero except the fourth one which should be one but because the untrained neural network just puts out some random values we need to tell it what output we expected so our three is transformed to this binary vector which is then used to calculate the difference to what's the output but more about that later just remember that the label is represented in a binary format with this in place we can now look at how to train the neil network the training occurs inside two loops the inner loop iterates through all image label pairs while the outer loop specifies how often we iterate through all images this means if the variable epochs is set to 3 we go through all images three times so everything i explain while we're inside these loops occurs three times for each of the 60 000 images if we take a look at the shape information for the variables img and l we can see that both are vectors this is a problem because we are doing matrix multiplications with the weight matrices later on and the operation fails if one operand is a matrix and the other a vector that's why we need to reshape both vectors with the following two lines the first line changes the shape of the variable img from a vector of size 784 to a 784 by 1 matrix while the second line changes the shape of the variable l from a vector of size 10 to a 10 by 1 matrix this brings us to the first training step called forward propagation it is used to transform the input values into output values to show this on the small network let's take 5 pixel values as input the values are normalized into a range of 0 to 1 meaning that a white pixel has the value 1 a black pixel has the value 0 and a grey pixel is somewhere in between depending on its grayscale to get the hidden layer values we need to take the input values and the weight matrix that connects both layers then multiply them through matrix multiplication and add the bias weights let's illustrate this in detail for the first hidden neuron each input value is multiplied with its weight connection that goes to the first hidden neuron the resulting five values are then summed up last the bias weight is added and voila we have the hidden neuron value note that the bias neuron is not directly present in the implementation because 1 times the bias weight values equals the bias weight values but it's more tangible to think that there is also a biased neuron as shown here you might wonder why the variable is named h underscore 3. that's because we are not done with the hidden layer yet the value in one of the hidden neurons could be extremely large compared to the values in the other hidden neurons to prevent this we want to normalize the values into a specific range like we did for the input values this can be done by applying an activation function to it a commonly used one is the sigmoid function it is defined as follows looks like this and normalizes its input which is h underscore pre in our case into a range between 0 and 1. that's exactly what we want we then repeat the same procedure to get the output values and therefore finish the first training step the second step is to compare the output values with the label which is zero please remember that we use a smaller network for the visualizations meaning that the network shown here can only learn to differentiate between the numbers 0 1 and 2. to compare the output values with the label we need some sort of function again this time called cost or error function like for the activation function there are many possible functions we'll stick with the most commonly used one which is the mean squared error it works by calculating the difference between each output and the corresponding label value then squaring each difference followed by summing the resulting values together and dividing it by the number of output neurons the resulting value is our cost or error depending on which word you prefer the second code line checks whether our network classified the input correctly for this we check which neuron has the highest value here it is the first neuron so our neural network classified the input as 0. because this matches the label we increase our counter by 1. if the label would have been 1 or 2 we would not increase the counter please note that this line is not important for the training itself but we do it because we would like to know how many images are classified correctly after each epoch now that we have the error value we need to know how strong each weight participated towards it and how we can adjust the weights to have a smaller error when we see the same inputs again this is the most crucial and complicated part about training neural networks the underlying algorithm is called back propagation you've probably already seen it mathematically written somewhere if not there you go but please don't panic rather look at the code it's actually just six lines backpropagation works by propagating the error from the end back to the start we start with our weights that connect the hidden layer with the output layer in the first step we need to calculate the delta for each neuron normally we'd need the derivative of the cost function but thanks to a few mathematical tricks that can be used for the mean squared error cost function we can just write o minus l so the delta for an output neuron is basically just the difference between its output and the label so what's with the error value we calculated in the last step then well we don't need it but i still wanted to show it to you because it is required when having a different cost function in the next step the delta values are used in a matrix multiplication with the hidden layer outputs to get an update value for each weight connecting both layers since the update values just represent how to improve the weights with respect to the current input we want to adjust the weights carefully therefore we multiply them with a small learning rate but why is there minus in front of it well i won't go into detail about it in this video but you can think of the update values as values representing how to maximize the error for the input so we need to negate them to have the opposite effect alright so now we have updated the weights between the hidden and output layer except for the bias weights the idea is basically the same with the difference that the bias neuron value is always 1. since there's no need to multiply something with 1 we can just multiply the delta values with the learning rate and negate the result if we look at the update for the weights connecting the input layer with the hidden layer we can see that nearly everything looks the same except for the delta calculation that's because this time we can't use some mathematical tricks to simplify the equation so we need the derivative of the sigmoid function h which is sigma times 1 minus sigmoid so we can write it as h times 1 minus h then we need our updated weight matrix transposed matrix multiplied with the delta values and finally multiply that result with the derivative values the resulting delta values show how strong each hidden neuron participated towards the error those values can then be used to calculate the update values for weights connecting the input with the hidden layer and if we would have a few more hidden layers with a sigmoid activation function we just have to repeat those steps over and over till all weights are updated that's it now you know how to train a new network from scratch let's run it and see what accuracy we can achieve while it's running i'd like to let you know that any additional information and corrections that might come up after publishing this video will be added to the description so if there's anything you're wondering about i've probably already added it in there if not feel free to ask in the comment section wow over 93 percent that's quite good but there's one part left what is it for well using the neural network in action of course let me quickly go through what's happening here first we expect an input between 0 and 59 999 which is used to choose one of our 60 000 images we then extract the specified image and add it to a plot object next we do the forward propagation step to get our output values and set the title of the plot to the number of the strongest activated neuron then we show the plot and can see that the neural network correctly identified the three so we scroll down hit the subscribe button and ignore the notification bell which is a huge mistake because then we cannot be the dude writing first or second in the comment section the code explained here will be available for everyone link in the description the video is animated using python a second video about how i created this video as well as the python source code for all animations can be accessed by becoming a patron link in the description thanks and i hope to see you in the next video [Music] you