Print Initialized Zip Code with airSlate SignNow

Print initialized zip code

hey guys this is shrini and in this tutorial let's continue our discussion about hyper parameter tuning in the last video we looked at how to perform a grid search for the best parameters for learning rate and momentum and in this video let's actually look at activation functions and weight initialization and optimizers what are the best ones what are the best activation functions what is the best way to initialize your weights and what is the best one for optimizer no one has an answer to your problem because you know it the best so one way to find out what is the best one is to go ahead and learn everything about all of these and then based on your knowledge say this is the best one for your problem because of blah blah blah whatever that thing is or do it in an empirical way which is what we're trying to do meaning throw a whole bunch of parameters at the system and say find the best one for me okay and then it'll tell you that's exactly what we're trying to do here and we did that in the last video and let's continue exactly the same topic so i really really urge you to watch the last two three videos on this topic so you know uh so you have some context about what we're talking about here so we're going to use exactly the same example of amnesty classification and quickly tweak these uh you know the the hyper parameter space of activation weight initialization and optimizer and then find out the best one okay so let's jump into the code right now and uh here like i said we are going to work with mnist data and all of these libraries are pretty standard so let's go ahead and run this again except for this one learning rate scheduler i don't know why we have this we are not using it this is left over from one of our previous videos so probably ignore that for now and let's go ahead and run our numpy and matplotlib and this is the important one grid search cv again this is scikit-learn model selection designed for scikit-learn models but we are going to use a trick to convert our neural network model keras model into something that scikit learn understands so the version of tensorflow and keras that i'm using are 2.2 for tensorflow and for keras 2.4.3 okay now the next step we are going to fix the random seed because we don't want the results to change by that much when we change different parameters because we want to compare those okay and we are going to import the amnish dataset talked about it in the last video and i don't need to plot because you know what it is and watch my previous video if you don't know what it is let's go ahead and normalize the result the x values otherwise the values are between 0 to 255 and we are flattening or we are actually reshaping our 28 by 28 pixels or 60 000 images 28 by 28 into a one dimensional vector so if you look at variable explorer now instead of 6000 by 28 by twenty eight we have sixty thousand by seven eighty four again something we discussed in the last video now we have to convert our y values into one hot encoded so we are going to use two categorical so because we have a classification problem right here and then uh in the last video again we discussed about this where i said well working with 60 000 and changing all these parameters can be very time consuming and probably unnecessary so what if i get 6000 uh random ones and the way i'm trying to do is i know we have x train so let's take that and of those let's get 90 assigned to x do not use we're not going to use it we're going to throw it out and x grid is something where we are actually going to use which is the 10 percent okay so this is i'm just using the scikit learns import train test split which we have done in many many videos in the past okay so this is where the advantage of becoming my subscriber uh lies right you'll know all of these and you'll be up to date with little things in here okay so uh the shape of our grid is 784 because i'm doing this because that needs to be part of my input dimensions right there here comes the actual model again i it's worth repeating from last video right we are defining our model as a function because this is the function that gets uh that gets supplied into the into the model later on so uh so so this scikit-learn understands exactly how to interpret it okay this is the wrapper that keras tensorflow actually provided us okay so that's why we are defining this as a function and the parameters of the function are in this example since we are trying to span activation initial weights and optimizer i'm going to provide these three as inputs and pre-define it to do something if you don't do this and if you just put activation equals to something here it's going to throw you an error saying that the model is not complete or some weird error so this is the this is the technique since these three are the ones that we are trying to span so these are uh provided right there and then we are building the model like normal this is exactly the same model as last time in the previous video and the one before that okay and we have a couple of dense layers each 64 neurons and a 10 dropout in the middle in the next video let's actually parameterize or span multiple hyper parameters including dropout epochs and all that stuff okay but for now let's just focus on this okay my activation uh in this dense layer equals to activation well initially relu but then we'll change it kernel initializer what is that how do you want to assign the initial weights randomly right so in this case i'm saying okay uniform random okay so that's what the initial weights are and then we'll change this and here again same thing initial weights activation and in the final layer i cannot just use any activation i have to use soft max this is a multi-class classification problem if this is a binary then that would be something else right a sigmoid basically so this is fixed this is not something i'm changing right now okay and finally like here the loss is categorical cross entropy obviously this is a multi-class problem and optimizers is optimizer what is it we can try adam we can try something else okay a stochastic gradient descent and a couple so this is uh my model so let's go ahead and run these lines there you go now we are going to import keras classifier from keras.wrappers scikit-learn so we can define or we can define our model such a way that uh scikit-learn understands okay that's the whole point so let's import that and for batch size let's use exactly what we have done the last time and let's do 10 epochs because doing 50 epochs where all of these can take quite a bit of time so let's just do 10 epochs and now my model for psychic learn to understand is keras classifier the one that we just imported and my build function is define model which is this one okay so build function is basically what model we are trying to supply number of epochs is 100 batch size is batch size everything is pretty much the same now comes the dictionary part so what goes into this grid search cv is our model which we just defined and the parameter space that we want to kind of explore as a parameter grid this is nothing but a a dictionary okay so first let's start with lists what do i want to do for activation i want to do these three softmax relu and sigmoid tell me which one works best you can also try tan h and soft plus a a couple others right there i just left that in the notes how do you want to initialize your weights you can use uniform normal he uniform which one works okay and again a couple others that you can try uh down here for optimizer do you want to use stochastic gradient descent rms prop or atom okay so this is the space nine of these that we are spanning uh again it's going to use cross validation to actually see how the results are you know between these and then what is the best combination and it's going to tell us what that is now i'm going to convert this list into a dictionary using the dict command right there activation is activation initial rates and optimizer so these are the three things so let's go ahead and run them i don't think we did did we do uh no after batch size we did not run this model so let's go ahead and run all of these just to make sure okay so now we have our parameter grid and if you want to see this is how it looks like a grid a dictionary of three lists that's pretty much it okay and let's go ahead and define our grid search so grid is my grid search cv again the inputs are my model and the parameter space i would like to uh include number of jobs equals to 16 again i mentioned this in the last couple of videos but it's worth mentioning i have 32 cpus if you're working on a laptop you may have two or four depending upon how rich you are i should say this doesn't mean i'm rich someone was throwing a workstation out three years ago that he or she purchased two years prior rich person i begged to get this workstation so i have it it's old but it works it's got a couple of gpus old gpus but they work okay so um and out of those 32 my number of jobs equals to 16. i'm going to use only 16 of them and the cross validation i'm using threefold cross validation you can change it to five if you want okay so this is the description of my grid and now i just need to fit it to what to my x grid and y grade right so our x values we have six thousand y six thousand values so let's go ahead and fit this this may take a few uh a while first of all let's fire up this so we can see the action okay it's always nice to see all of these going on so you can see how things are actually working so you can see how it's uh it's using whatever my definition of 16 is all of these are fired up it's actually assigning you know 50 of the threads over there and once it's done let's have a quick look at the results okay let me pause the video uh this may take a couple of minutes actually it's done it this is the advantage of having 16 cpus man i'm telling you i tried this on my laptop with a couple of a couple of these and it took forever and i killed it and now i fired up my desktop it gives up too much heat but it's about uh you can i don't know if you can i don't think you can see outside it's uh pretty cold in northern california almost like 34 degrees right now well in fahrenheit very close to freezing okay let's get back let's focus okay so once you do this the grid result stores a whole bunch of stuff it stores best score best parameters uh cv cross validation results that you can unpack so let's unpack one at a time so let's just look at the best score and best parameters so after all of that let's expand this so it makes it easy so the best cross validation score that we got is 92.6833 percent and using an activation function of relu and initial weights uh using initialization of weights using he underscore uniform and using adam optimum these are all my favorites when i'm actually putting together a model i don't know any better i always use relu as activation i use he uniform and then i use adam that's it atom is adaptive uh you know when it comes to loss learning rates sorry okay so that is it and if you want to look at all the other details how it performed with everything else let's run these lines of code and expand this so you can see everything here so you can see starting 11 this is 48 using soft max and uniform and adam so again you can study it at your own pace on your own problem so in the next video let's actually continue this using the same example except let's do four uh five different parameters batch size epochs and a whole bunch uh and then and then have a quick look because uh there i don't think you need to any anyway i'll i'll save that for the next tutorial it's going to be very similar but please watch it because you may learn something something new over there and if you have great computing resources especially on google colab you can actually throw a bunch of these and for paid google collab for the free one you get only two cpus if i am right and this is not even used in gpu so it's not gonna buy you much okay anyway let's meet in the next video until then please go ahead and subscribe to this channel thank you