Sign Arizona Banking IOU Fast

Industry sign banking alaska iou computer

hey guys this is shrini and in this video let's discuss the best way to evaluate semantic segmentation and obviously we're going to use intersection over union approach now i hope you know what semantic segmentation is if not you are wasting watching this video now just a quick reminder semantic segmentation by that we refer to classifying individual pixel rather than classifying an image as a cat or a dog but in this case we are classifying individual pixels that belong to a cat or a dog in this example i'm just showing you a rock sample showing different minerals in the rock but this is what semantic segmentation is now why what's wrong with accuracy right i mean we do scikit learn dot metrics and from metrics we normally import our accuracy that actually looks at our prediction and our ground truth and then gives us the accuracy but the problem is it uh it's not a great metric if you have a class imbalance which happens when you have multi-class problems in real life so inaccuracy in minority classes okay if you have these minority classes any inaccuracy there they're completely overshadowed by the accuracy of the majority classes for example in this image look at this if you look at this image all you see is blue and red these are majority classes so if i get 99.9 accuracy on these two classes and if i get 50 40 accuracy on these minor classes overall the accuracy may be 95 but that doesn't necessarily mean we are getting good accuracy from these minority classes hopefully that makes sense yeah now let's look at what iou is and how it can help again i apologize for this slide because it's got too much text uh at least i get bored when i see these so let me quickly go through this iou by definition is intersection over union intersection is basically overlap between your prediction and ground truth right and union is union between your predicted and ground truth now if you want a keras definition it's basically true positives okay which is a class that's uh labeled you know as for example two and how many of those are you getting it correctly okay so true positives or a class that's labeled cat how many of cats are you correctly labeling as cats divided by true positive plus false positives okay plus false negatives how many dogs are cats and how many cats are misclassified as dogs right so when you add all of that it's union so that's the keras definition now it's customary to report like a mean iou or an average iou in fact when you use the keras built-in function it does report the mean iou but it's very important to look at individual class ious because it does tell you something about the accuracy of that class and i'll show you how to do that using a confusion matrix and don't be confused it will be very quick tutorial now for this i'm going to use the code that we have already used in my video 159b where i talked about using pre-trained convolutional networks specifically vgg 16 i believe not vgc 19 but yeah vg-16 and uh uh taking vgg-16 and taking only the pre-trained weights and using that as feature extractor sending our images and labels through sorry images through it extracting the features and then using a random forest to classify it please watch that video if you don't know what i'm talking about and i'm not going to go through this code again only the part i'll go through is the iou part now what is our problem in this case so here i have let's delete this segmented i was testing the code uh let's just keep our test one dot uh image right there this is the image that's actually we are trying to segment and the ground truth for this image is right here okay so this is the image we are trying to segment this is our test image by the way and this is the ground truth so i'll train a model and use that model to segment the image on the right and then take that segmented result and compare it against the ground truth using our iou that's the plan for now okay and all the testing i maybe i'll just show you uh i have training images a couple of training images and a couple of training masks okay and the masks not much not not many labels very few labels for the mask if you want to draw your own masks for semantic segmentation if you don't have a existing tool you can go to www.appear.com a-p-e-e-r it's free so sign up go ahead and do this that's how i do it i did this you know this specific example here okay so now that you have good backstory the whole part here is just reading the training images reading the training masks getting the data ready and uh getting the vgg model only the weights that got trained on imagenet and using that as feature generator okay and down here when we come down i'm going to uh let's get down let's get down further so i think i got way too further down so let me go back one step yeah there you go i was looking for this random forest classifier so uh we're we're basically classifying the features that got extracted out of uh you know out of these images and then fitting a random forest again please watch 159b i sincerely apologize for speeding this through this is not my teaching style but but i really want to focus on the main content of this video which is why you probably tuned in anyway so the assumption is after all of this you have a training image and uh a model okay or a testing image and a model so now uh i already ran these lines so i don't waste your time so let's directly get down to the par part where it makes sense okay so what makes sense now i have model i trained model in fact i saved that model i'm loading that model okay now i want to test the accuracy first thing first let's go ahead and read the test image if you remember the image i just showed you resize it obviously all pre-processing converting rgb to bgr because that's what i did for my input images and then expanding the dimensions to make sure the inputs to my prediction to my random forest uh uh are compliant but before sending it to random forest first of all i need to predict it to extract features yeah new model.predict this is basically extracting features and i'm reshaping it and i'm uh and i'm performing my final prediction using random forest what is the loaded model the loaded model is the one that we trained on now i'm just talking about the way i did segmentation you probably have done your segmentation using otsu very simple binary segmentation using deep learning fully convolutional or for you know any other approach unit for example it doesn't matter get to this point where you have a segmented image and your ground truth okay and i'm just showing you the way i got there that's pretty much it okay and then i saved the image now now that i have my segmented image in fact i may not have done this step so let's go ahead and do this in fact i did that but i removed i deleted the file in front of your eyes so let's go yeah there you go so i have my test one segmented image this is my segmented image and if you remember this other image test mask versus this this is our segmented image and this is our ground truth obviously the ground truth is very clean and the segmented image is kind of crappy in here but let's see if we can mathematically extract or you know extract some quantification some some values to prove that okay so first of all uh i'm going to also load my ground truth image because we're going to compare that in a second and normally what would we do we just do accuracy right you just say okay from scikit learn import metrics and from there accuracy score go ahead and predict it i'm curious okay you're getting 94 percent we're done right so normally 94 is pretty good um maybe you increase the number of trees here from 30 to 40 and then see if it affects your accuracy which it does but let's not let's not do that that's not the goal finally after eight minutes or so or nine minutes we are down to talking about iou okay so i'm going to use iou again you can write your own code because you know what i'm talking about right you can use logical and logical r from numpy to see if you can you can do intersection over union or you can just create your confusion matrix and then get these things but anyway i'll just show you the keras approach so from carriage.matrix import mean iou okay so this is what we are going to import right there and then number of classes is five zero being or background class i don't think i have any pixels with a value of 0 in my image but that's fine let's include that and the way you instantiate your mean iou is basically mean iou and how many classes let's go ahead and put five classes in there okay so that's how you start it and now we need to update it update it with what with the truth and prediction right so or our ground truth versus what we predicted so let's go ahead and do that and that should give you a tensor you see here so this is all my pixel value zeros obviously i have no pixels with zero so i'm not getting anything here the next one is a pixel value of one and the next one is a pixel value of two the next one pixel value of three and pixel value of four yeah i have four pixel values one two three four zero being the background so as you can see majority of these are classified as correctly classified i see wrong classification there wrong classification here but let's quantify why why speculate okay so the best way first of all if you want to get a mean value for iou let's go ahead and select mean iou and the way you get that is once you have your iou.karis updated with this you can just do dot result then it will give you your mean iou so let's print it out and it gives us 77.9 already you can see a big difference when you do accuracy it's telling you 94 when you do mean io it's saying 77.9 that should tell you something is wrong okay or i mean the accuracy is not as high but if you just look at accuracy you would be happy with 94 this is exactly the reason why iou can be very very useful now if you're curious about which which classes are doing good which classes are doing bad now let's actually get down to uh now that we have this right we have this matrix all we got to do is take your true positive right your true positive divided by true positive plus false positives okay and false negatives so i just wrote these lines of code that actually does exactly that i think i did a good you know did it the right way but if any of you find anything wrong with this let me know but first thing first i'm converting this tensor into a numpy array so let's go ahead and convert that and now you can just print the values so you can just see the values here well let's actually go ahead and open up values so i don't like this 10 to the power of 5. i'm not good at that but anyway you can see how my class number one okay most of these are correctly classified as class one class two thirty nine thousand nine twenty three and misclassified seven thousand there and thirty two thousand so this is pretty bad visually you can tell that class two is not doing a good job because nearly half of the pixels that are supposed to be class 2 are misclassified as class 3. my class 3 is doing pretty good 422 000 pixels you see how the majority classes are doing pretty good even though 3 000 some of these are off that's fine and class 4 is also doing pretty good 25 000 and the reason why class 4 is doing good even though it's minority classes this is very bright pixels it's easy to segment it so it's doing good this class which is class one or two one of these majority classes is also doing good because it's majority the black region is also doing good the class number two which is these gray is not doing good because it's tough to segment those uh in reality so visually we have an idea and now all i'm doing is exactly doing this true positive divided by true positive plus false positives and all that so let's go ahead and run these lines and oh i forgot to include print statements which is not good um let's go ahead and open it up here okay so class one i o u is 0.963 96.3 for class 1. great class 2 46.4 very horrible accuracy as you can tell right and class 3 is 89 not bad and class 4 is actually 77.9 so this bright phase is class 4 it's only 77.9 this dark phase which is is 44 and these two are in about 90 or 90 plus percent right there so what's wrong with this well my class 2 i have to do something about it what do i do in this case i label more images with class 2 being represented more okay you can do some sort of balancing you know you can kind of increase the number of class twos by following one of the methods i talked about like smote smote or one of these but you do that as a last resort if this is the case in reality 46 of accuracy but others are doing good this is a classic imbalance problem so get 10 more images but only label the ones that you want and increase the number of pixels that represent that specific class but if you cannot do any of that then think about balancing the data by either under sampling the majority class or oversampling the minority class i hope you guys found this tutorial to be useful and again in the next video let's cover something something different something useful something fun thank you very much and please do subscribe to my channel thanks