Print Initialized Gender with airSlate SignNow

Print initialized gender

so I'm Jeremy so I work at Neil I'm gonna apply research scientist there I have a background in engineering physics and then in the last few years have really been focused a lot more on everything machine learning and deep learning related so Neil I itself is a research institute that really focuses primarily on anything deep learning so it's really like it artificial intelligence Research Institute but I would say 95% of what goes on there is probably deep learning based how I was introduced to this project yeah had contacted me hi everyone if I can just say a little something to talk about the clinical implications before Jeremy gets into the heavy AI stuff so what we were trying to do is try to create a tool for a transgender patient for gender recognition of the voice in the clinical setting after treatment for transgender surgery so that's kind of our clinical standpoint and that's the reason you started working working with Jeremy's gonna go into that the heavy explanations here and if anybody has any questions I'm happy to answer thank you Jeff it's called what indeed so the objective of the project is to build a AI based tool to differentiate between male and female voices and the context is to be used to evaluate the success of voice therapy in an objective and quantitative way for the transgender population so yeah it has sort of briefly mentioned what the problem statement is about and she could probably give more information as to like something some more of the details I'm gonna go more into the technical side of what it is that we're building to actually have some quantitative measures to give a bit of background there's not really any good objective outcome measurements in terms of success for when someone goes through the changing of voice in the transgender population so there's a lot of subjective measurements and this is where I'm not an expert but if I understood correctly people sort of evaluate how they think they sound compared to and they have all sorts of tests and there's also other measures that will use various and there is parameters from the recordings but what we kind of want is just have an end-to-end tool that takes an audio clip and gives a quantitative number that says how confident is this prediction that the sound clip that we listened to was male or female and so the clinical implication of all this is that if we have enough data so we record a bunch of people say typical male voices typical female voices and we train a system to identify between male and female voices then if we know that that system is robust on male and female voices when someone goes through change of voice they can then measure quantitatively like how does this system think compared to all the voices had seen in the past that this person ranks in terms of a male voice or a female voice we're in the dataset that we're working with is a curated data set that yet has obtained for us so we have approximately 300 voice recordings and they're all of high quality they're done in a lab lab environment where people are asked to talk in a specific script and there is all sorts of different cues and they they'll sweep their boys like going on different things like that so we can kind of hear the all the different sound qualities of their voice in the audio file and each sample is approximately 30 seconds long but that kind of depends on where was recorded so the gender breakdown that we have in this data set it's slightly unbalanced so we have more female voices than we have male voices and also one thing to note is that it's not particularly big data set it's 300 voice recordings and in the context of training deep learning models usually you want in the thousands or millions of examples but we'll address how we kind of overcome this so the high-level approach what we're doing is we're doing supervised learning using CNN so CN n stands for convolutional neural networks on voice spectrogram so what that means is we start with our voice file actually I have a loaded voice file here but you won't hear it if I click play because I'm bring through headphones so I won't actually play it but this is a sound file that you can open on your computer what we do is we feed it through a speech recognition library in this case just as a detailer that we use The Grocer but there are a few different libraries that can that can go from sound file to spectrum so we convert this sound file into a representation that's two-dimensional and what we have in the spectrum is we can think of this axis as time and this axis as frequency and the color here is the intensity of these frequencies so as people are talking the various frequencies that they emit and and what we then do is we take this spectrogram and we feed it as an input into our convolutional neural network so here we're using the pipe torch framework and we have a whole training pipeline setup and the idea is that once we feed this through our convolutional neural networks we can then get our prediction of the percentage confidence of it being a male or at being a female so diving in a bit more into the details of our implementation what we decided to do is oh so maybe I could speak a bit more about the spectrograms so this vector guys themselves I mentioned this already we use the grossa it's a open source library that allows us to generate these power spectrums and there's actually quite a lot of parameters that we can play with so I invite you to go read the documentation or maybe talk to me afterwards that you'd let them know then what about this parameters but some of these parameters are the hop size the window length how many male bands do we want here so when we're sampling and taking a slice of a spectrogram how much of the voice are we actually including and a lot of these parameters could actually impact how these what the spectrograms look like so during training what we do is instead of giving the entire spectrogram so this entire spectrogram might represent approximately 30 seconds of sound what we do is we'll we'll take a random slice out of it and so every time we iterate through the data set we'll take slices at random and to this slice we always know that this whole spectrum is associated to the same gender so particularly here we take a slice of 256 by 80 and these are just implementation details so this can be thought of as an input matrix or an input image if you will tour CNN because cno's are typically used with images so we take a slice and then we'll feed that through a network then we can take another slice and feed that through our network and the reason why we do that is this is how we sort of augment our data set so we only have 300 of these spectrums but from these 300 spectrums we can sample many many different slices like this and they all very rarely overlap reviews they're very rarely be the exact same slice so it'll just it's kind of a trick that we use to argument how much data we're actually giving to our network and it also makes it makes these models a lot smaller smaller and easier to train and it makes it such that we can then evaluate on the entirety of the network so while we're training we only go one slice at a time oh and this is another utility for that that we are accounting for the disproportion in male to female in this data set and so while we're training we're going one slice at a time but when we're validating and testing our system what we do is we then take all the different slices that we can take and take the average main prediction of all these slices so during training we're just saying here's one slice give the proper prediction but then when we're in valuating we're actually going to go and sweep the entire spectrum and say for this patient that's never been seen before for all of these different slices here is my prediction and then take a mean of this prediction and hopefully the majority vote is the actual gender that we're trying to look for I see you know the left side of the spectrogram looks like speech and the right side looks like a vowel or constant phonation on a pitch it really doesn't matter to your machine that the tasks are widely varied no in this case we're trying to generalize across all this actions which is also kind of the strength of this approach is the more varied data we have the more robust this system will be the downfall to this is that sometimes you might sample an area where say someone is not talking a lot and that can be like what will be the difference between say your system predicting always super confident and having some kind of uncertainty because you might be sampling from an area where maybe there wasn't a lot of speech to begin with talking about some preliminary results I want to emphasize that they're very primitive in area so far you know we have the whole pipeline set up we were able to train and evaluate and we're getting approximately 90 to 95% accuracy on all the training validation and test sets which is very promising but this is sort of our first iteration and we still have a lot of work to really appropriately give some validation measures so one thing that I would like to do for example is this series of cross validations and also sampling from potentially other data sets that are out of distribution or not necessarily these specific recordings just to see how our system is doing so this is kind of our second point we just recently finished putting proper evaluation before the validation and test sir and the test pipelines and now we're at a point where we need to both improve our models and get better metrics to really understand both on a per patient level and on a spectrum level what's happening what when our model scaling are they failing because those voice recordings are very hard or are they failing because there's something we're failing to capture we're still in the process of figuring a lot of these details out to translate that 10% of the time five to ten percent of the time it's calling a man a woman and a woman and man or it's saying I don't know what it is or both so that's a good question so we'll see in the demo sort of how our evaluation tool works but so we have this majority book so the thing is that our our vote well the final result will be male female and you'll actually be able to see the confidence for each so if you going back to this slide here I'm taking let's say let's say four or this for example say that I'm sampling twenty spectrums for patients so twenty times I get a quote now if nineteen out of those twenty times each one of these spectrums gives me like ninety percent then I'll be ninety percent confident that it's a male more or less but if throughout this whole thing let's say it's a voice that's really uncertain I'm very unsure then my final vote might be for female but it might be with only like 60 percent accuracy or 55 percent confidence over all these votes or it might be just like a luck but in the at the end you still have one results coming up but we're giving this metric as well like how much on average did you score I don't know if that answers your question but I'll also show an example of that so one nice thing that we have so far and keep in mind that this is very much preliminary and a proof of concept for that we put up a live demo that you can go and use online so the link is in the slides here you can already access it and I put just a few slides to show you how it works although it's all the information that you need is self-contained it's just if you've never used this particular platform before there's additional instructions just in case so we're using Google collab which for those of you who aren't familiar with collab it's a Python Jupiter notebook environment that you can run directly within your browser so what that means in this case is that we can write all the code and sort of abstract it and just make it very user friendly and you don't need to install anything on your in you can pretty much open this and as long as you follow the instructions things set themselves up automatically and you can run this on your own sound files that you would like to evaluate and this tool isn't perfect it's just sort of an example feel free to use it feel free to send us some comments and if there's anything that's unclear let us know so I'm just gonna walk you through how this works with the sort of few samples and then I'll actually open it in my browser fire it up and show you what it is you should expect when you read this so this is the page that you see when you first land on this link you should probably be signed into Google when you think you this is a Google service so if you're not signed into a Google account I'm not sure if it'll work and so the first thing that you want to do is you want to hit these play buttons so here when you hover on top of this you'll see a play button so you want to first initialize this workstation so you hit this play and what it'll take a bit of time to initialize that's because it's going to go fetch our code and fetch our model it's going to install everything locally also Google is going to ask you if you're sure that you want to execute this code because you are executing code from a random stranger so you should execute this will make you feel comfortable running this and then once you're done it'll say it'll say done and that means that you're done step one which is to initialize this sort of setup the whole environment then what you need to do is click on this icon that you'll see here these instructions are all going to be in the collab and once you click on this icon you'll be proper you'll be able to drag-and-drop files so you'll be able to upload audio files into your workspace and so what's that what that'll look like is you know sort of like your standard driving drop your WAV file will appear here and once you see that your file is properly added you can then hit the play button on the analyzed voice section and then it's going to start running and once it runs it will give you the it'll analyze the voice the WAV file and it'll tell you the probability that it was a male voice and the probability that it was a female's voice and in this case this was a actual email example so maybe I can do a quick run-through over this right now just to see although live demos are rarely a good idea but let's try it anyway so I'll just refresh this just so we're sure that I'm not cheating and so I'll put this in English for some reason it defaults to French with me so first thing is we do is we initialize our workstation so here it tells you this was not this is not code that is sanctioned by Google it comes from github so you know feel free to go from github look at the code before you run it so in this case I'm gonna run it so this might take a little bit of time I think it takes about maybe 30 seconds to maybe a minute where it downloads all the code see so that I didn't so first it actually asks Google for a machine Google gives you a machine then it goes starts downloading all the code puts the models where they need to be and then when it's eventually done it'll just say done so let's just wait here for a little bit I can take some questions as well in the meantime if anyone has questions ok so now we see here that we're done so we can then go and upload our files so it says here click on the folder icon so here we click on the folder icon then you just go to your computer you go grab whatever voice spot you want to grab so we'll just use the same one sees just drag and drop so it gives you a little bit of a reminder here so you just say ok so you just wait till it's uploaded now we see here so it's uploaded and then you can just go and click this cell here so analyze voices click click and you're gonna see a little bit of a status bar here so seeing that it's gonna start analyzing the file and then it should print the results once it's done I have a question while that's running yeah you were you actually have no idea of what is being analyzed because you didn't tell it initially in male and females what to look for it just collected the data and it might have looked at pitch it might have looked at all kinds of things spectrum but you just have no idea and so you're not you you've done nothing to selectively train it other than give it a file that's well so the one thing I have done is this determining how to actually collect this spectrogram which is the one piece of information that the machine doesn't figure out for itself is how to convert this file to this vector but once this gets converted then yes you're right I just say here are all these examples now you figure out what the best way to get the highest score on this data set would be based on only this information so then let's say Yael and her clinic records some sort of audio may be running speech and some vowels and then I record a completely different standardized speech and I asked the patient to hit their lowest pitch but you'll never did that my daddy will somehow be different than because we didn't do a standardized protocol yep that's great so this is like this is just deep learning in general there's this big hypothesis hypothesis that gets kind of pushed aside all the time is that usually you're sampling from an iid distribution so what that means is that you know you have you you make the assumption that your training set and your test sets are all being sampled from the same distribution when you're out of distribution which is can happen this will depend on how much data you actually fed your model to generalize to just all sorts of data that can be so you know for example if you and yet our recording similar data but not necessarily the exact same speech then there's a good chances that it will generalize but if you then take I don't know someone's singing a song it's never actually looked at people singing so that might confuse the network so the more data that you have the more varied your data is the more robust it becomes because you can evaluate it on more sets and yeah it really just learns to find the optimal set of parameters to extract from these spectrograms based on the data it's been trained on is any kind of like quality check for your acoustic data kind of like is there any kind of like noise harmonic ratio or the view like record the sounds in a soundproof box so the data set we trained the computer with it the data set from Patrick Walden he's a PhD speech therapist at collected normal voices from different laryngologist so so basically there he all standardized there were six different labs across the US but not standard US and what Jeremy told us before starting the study is that it was even more interesting to have data that was not necessarily standardized to better train the computer as he said to before so when you want to look at vocal pathologies for example you'll probably want a more standardized way to collect the data what I have to tell you is that all the data that we use we eliminate all biases like somebody else talking so the most important bias of course is if the laryngologist is talking and asking questions so that was all eliminated so that important bias were eliminated 200 before recordings and 200 after recordings on patients at surgery and I could go to your collab research Google and just run parts of the recording through or all of the recording through it and see what it says out of interest I guess that yeah I said so that's the second part of our study we're working on right now we're actually working with a center to run the transgender population through that set and we're correlating for for other stuff like fundamental frequency and vhi score so that's the study were conducting right now to see if there's a correlation with the outcomes that you know exist already now this was the standard set you haven't actually run any of the transgender patient voices do it yeah correct right now we've only ran normal voices and this is what we're doing right now we're starting the study for the transgender population yeah just keep keep in mind though so that this is kind of like our first iteration so it's possible that it doesn't work great right off the bat we haven't really validated it on other data other than the data we had so it is possible that it's not perfect at the moment but we're going to be refining it as we go yeah you can definitely feel free to use the collab and if you do get some some results maybe just let us know like that would be a good data point for us that this worked or the didn't work at all it would be kind of interesting and then you know do you like when I do my recordings I have a mic on the patient but I'm across the room so my voice does get into it you take your recordings that go in and slice out any talking or you just have perfect recordings or yeah so for the data where we're using for transgender patient we're slicing out any time the physician is talking where the SLP is talking for sure so that's that's the data we're using or that we suspected Neos as well we're using retrospective data and furtive data but we're any any other voice some time did you do any changes in volume on your standard set or did you do also highest and lower you in other words as a human if I come up to a person who's in a booth and I want to know if their mayor of female 1 the test I might do is to ask them to hit their lowest note and that would be a huge cue much more in conversational speech if I just know the lowest pitch that anybody can hit that one piece of data is really important but you don't train these programs on that one piece of data it takes the whole spectrum so if you don't mind I'll just answer it this one for a clinical reason so I think what we're trying to do is is build them an outcome measurement for transgender for the transgender population to see if they're recognized as a male or female when they speak which is the most important psychological outcome for them and when they speak usually they speak at a comfortable pitch all right and that's why we're measuring on vhi as well on the tvq questionnaire is do are they perceived as male or female during conversational speech because obviously they don't they don't talk to people trying to reach their lowest pitch or highest pitch so that's why the data we use is at comfortable if I could just add something quickly dr. Thomas if if you'd like to upload any audio files it's better if you do it completely independently from what yeah I'll give us because I'll be a better indication to if generalize to other samples from somewhere else in the world if it works on your data set as well it's even better so it's better if you just put it without any transformation record the person if you want to do the low note do it and try to upload it and see what it does because this is going to help us make it super or best to handle edge cases actually I just need to add something really quickly if anyone wants to try to play with the collab website just make sure regarding the extension of the audio file it's only WAV and mp3 so if you try to upload something else than those two extensions it's not going to work I think what time we'll do some quick transformation where you can upload anything but for now it's limited to those two extensions it sounds to me like if if you want to test something you ideally should then start with a data set a different data set if you're going to test different part of speech or low and high pitch you would want your standard data set include low and high pitch and then and and really score that with the machine learning and then giving your question questionable data set yes sir I ideally you want to have always like a well guarded test set because the the fear is that it's a deep learning models in particular are very prone to overfitting theta so the more you can have a robust properly guarded test set the more you can be confident that your model didn't just learn to cheat and you know like memorize patterns for example so the more varied our test sets or the more difficult these tests are the better our evaluation metrics and our confidence that this system actually works will be and you could then create a data set of having a person breathing air out and and recording the audio with varying degrees of breath leanness and clarity and it could start to without you telling it to look at high frequency noise or broad spectrum noise you could just say to it which or breathy and which ones aren't or to what degree of breath penis there is could you pick one parameter and set up a data set and then explore just that parameter yeah yeah absolutely so in the way that instead of this specific problem it's set up in a way that we're just analyzing gender so it's it's a binary classification but you could also do multiple categories that you had some so in terms of correctness I don't know what the quantitative measure is but you can also set up your problem in different ways you could set up your problem as a regression task which means that like instead of having a zero or one value you might have a range of possible values so between zero and let's say a hundred pick a number in between there any any possible number in there so it really depends on how you set up your problem but what to remember is that the main framework is given an sample what is the target that is associated and idea that you want as many examples as you can and as properly or evenly distributed as you can well thanks Jeremy and Yael for sharing all this with us appreciate it loser