Discover the Best Invoice Text Example for Security

Invoice text example for Security

[Music] we are going to talk about one of the products that we've been working on just for a couple months so it's titled invoice to vac there's a few things we're gonna we'll talk about what that means and but our path so our you know our path is h2o helping a client to create a I to read documents to extract document information so we'll be talking about that our team there's where there's there's four of us so Brandon and I have been working with a particular client with PwC who spoke earlier today for about two years I've been working with him for about three years Brandon for two and for this project we've stepped it up a little bit so we've doubled our team we have a couple other data scientists helping us out for this one and then for all the products we build of course you know it's really not just the data science team we have software engineers helping us out build the front end that you saw actually on the screens our stuff was behind him earlier in the day and the customer team too so there's a lot of people in fact one of them is practically on our team as well so it's a fairly big group for this one I suppose specially in h2o terms for making this product we've been working on it for about - about three months almost now for this one so we'll show you kind of where we're at you know we are not done where we're headed what the ideas are what we struggled with that kind of kind of stuff so quickly to look at the problem and then we look at it a little deeper later but we are extracting specific information from business documents that's that's what we're doing various business documents the information we're pulling out of them changes and the different types of documents but that's the general idea so we have an example of one here you can see an invoice on the left and there's in this one we kind of circled a few of the things we're looking at again even for a specific document we can look at it multiple different ways we talked about that a little bit but in the right is where we're really headed we're just standard old CSV we're looking at you know actually the formatted information that we need to pull out of it so it's somewhat standard tasks when you're probably familiar with other people are doing it so where will what we'll talk about here took a little deeper at the problem and the challenge is in those that we've run into and some that you know are just native to this problem from when we this Arden will jump right into kind of the high-end one so there's there's a lot of algorithms going on here we're using many different things to do many of the different tasks but the deep learning of the most exciting so Brandon's going to talk about some of the image recognition the bounding box identification type algorithms that we've been using for this task and time permitting we should be able to do a little bit about what we've learned how we've prepared the data for this task it's not a standard one when you're teaching these image recognitions it's not cat dog Mouse either so it's a little different so we'll walk through what we've done for that what we've you know where we succeeded where we failed already in these three months and then last we'll try to finish up with how this connects into drivers so the product you've been hearing about kind of all day we are using it for this product and we'll talk about that and then also the real goal here is that the results of our work can be fed back into the product that we can actually have some recipes out of this so we'll take a look at where we're headed there that you know quite a while away but that's where we're headed so the problem challenge again you know pulling information out of these documents you know that when I tell this to a lot of people they go back to isn't this a solved problem why aren't you using OCR why aren't you using XYZ library why aren't you just using you logo Yolo why is this a thing you're working on you know and it's true there's you know many off-the-shelf solutions do a really good job at doing some of this tasks and so for example we have you know on the top we have a document on the left of PDF that we would we would typically see it's not an invoice but and on the right and you know a free tool on the internet or the first Google search will give you something that looks really clean really nice like not just OCR where you're getting the text back is preserved all the structure it's put a lot of high-end features in there it looks quite a lot like that document on the left and that's for free so you know the state of the art for pulling some of this information is pretty good you know there are a lot of off-the-shelf tools but if you take a closer look and on the bottom we zoom in to what for us is not just a mistake but it's a critical mistake so we this has used an English language OCR and so in the balance column a lot of our data is tabular that's a little different than typical of see our typical NLP so there's a very we were looking at orientation that's important to our problem as well so here in the column of balance we see it's supposed to be euros but II you are isn't a word and any others USD to be fit be honest but but it decided that the probability of going that it meant ADA was enough that it just switched it you know so it use data even though probably read the EU are very well and in that's part of what you get with some of the OCR tools they have deep learning underneath them there they're a language specific you can pick which one you want it's not that we chose the wrong language this is in English but the tool has some pros and cons and for us there'd be a critical punch so we can't just OCR it although the OCR is really good on everything else just about in this document but so it and it's not just one tool either even if we could you know we could run this a couple different ways and resolve this problem and it we will actually but there's other things too and there's a lot of judgment that goes on I'll skip to the third bullet point in some cases we're interested in the way that this document would be accounted for gary spoke about how in 1400s we have double entry accounting so we want to see how this this document would be accounted and so dates are really interesting to us but not every date not every date on the page we might have seven we'd have a hundred dates and only one of the ones is the correct date so OCR will give us all those dates it'll give us the text we can read it but that's not the entire challenge we have to then go extract not just extract you know what that is in a nice format read you know the words we we do actually have to figure out which is the right one that's our task and that's why this is more into the machine learning realm at that point so now you're talking about a classification model trained on data and that's really that's exactly what we're doing here so there's other people doing that too even specifically but we have yet to see something that would do what we want to do right out the box where we would use it actually so technology as we anticipate being in our stack that we were experimenting with now certainly in many different ways and in series and loops OCR or natural language processing image recognition and of several different types too so that I think shops over to burn it all right so I'm going to talk a little bit about how we've been applying or experience we've been doing with deep learning models and doing computer vision on the documents the reasons is mark kind of alluded to we need to do that it's because although a lot of these documents have like embedded text in them a lot of texts in them is not that good so like sometimes there just be a bunch of gibberish and sometimes would be slight misspellings but we need things to be exact so we can match the exact words in the document to the accounting documents that we need to match it to and then another reason is that the embedded text is usually just one giant long string of text so you kind of lose all the structure so kind of like Clark said you might have a bunch of different invoice dates in a document but we need it to be like if the invoice dates usually probably to be somewhere at the top of the document maybe on the right side or right in the middle or something like that so what we do is we're starting to use object detection models to identify bounding boxes of where the targets are and the document so far we have experimented with three different frameworks we've done faster our CNN and Retin internet which have both shown some promise so far we've also tried Yolo v3 which hasn't been that great yet but we'll keep trying I'll talk briefly about faster our CNN and retina net since those two have worked kind of the best so far and they flatly different takes on how to find objects so fast our CNN is a two-stage model which so basically takes an image reapply a convolution or network to it which outputs the feature map and from that feature map we apply a region proposal network which basically finds an image all of the possible boxes that the foreground might be in in our case the foreground is going to be text so it a better generate probably a thousand and two thousand different possibilities of where a bounding box like me but around specific text so once you have all these thousands of possibilities a second sage applies a classification network to it and that classification network will identify something and say this is an invoice date with a 90% chance or this is a stock code with a 4% chance or something like that and then once we have all of those we can pretty much say everything above 80% or you're gonna keep that and anything lower than that we're gonna throw it away because it's not useful or it's probably wrong then the second one that we had a showing promise is Rhett net net this is this doesn't have Stu sages like frost our CNN does this is a one-state detector but the thing that makes this one unique is that they introduced a new loss function that hasn't really been used before and that helps the one Sage detectors one benefit of the bunch of stage detectors is a lot faster than two sages so this loss function what the Sox function does actually is it punishes basically all the easy examples a lot harder than it does the harder examples so something that's hard we're gonna the skin of forces the model to focus on all the hard examples so maybe if it's struggling with finding an invoice state it's gonna force the model to start focusing and finding that a lot more so this is an example of Retina net results that we have so far I think we started using retina at about three weeks ago so this is very early stages as you can see in the upper left there we found the invoice number sorry all the green boxes are the actuals and all the other colors or guesses for different things we can see in the upper left that it got invoice date or sorry invoice number pretty much exactly correct but right down there right next to it totally missed the invoice date and it totally missed the company name didn't guess anything there and then we can see down in the bottom there was a few false positives and one word just has a box of emptiness which it's kind of surprising but that should be pretty easy to fix but yeah so I should also say that these models were trained on using 14 or 15 different invoice formats which and terms invoices is like nothing there's thousands and thousands of different formats so considering are you so few so far I think this shows pretty good promise and I think marks gonna talk about the training set yeah so that what he mentioned is one of the things we've learned so for us looking at this from the outset training did it was gonna be critical because for these models we don't readily have answers you can think the natural process of figuring this out would say well let's get a hold of some documents and the way they were accounted for you know that's the natural way to grab data without trying to go do it specifically for your process but a couple of problems of that one is you know security of the clients say that's actually kind of hard to find that pool we're not the only ones in that but if you look there's some other people solving this problem they've reported similar examples of confidentiality security you know this it's important so you can't just go raiding documents all the time but the real problem is that doesn't actually help us for deep learning what that would do let's say we had that so we have an invoice we have an invoice date here again like I mentioned there might be 100 invoice dates and 50 of them might all be the right one for us we kind of want to learn the why that's what that's what deep learning is is we really wanted to teach it the why this is the right one not that what the answer is like the actual date you know January 3rd doesn't matter at all we wanted to learn the flexibility of the thing that follows invoice date in many different languages many different representations sometimes it doesn't say that at all you know that's the thing that we wanted to learn and so we've we've gone after these bounding box models and so you can look in the bottom right you know this is a spreadsheet view of it but this is the data we've had to collect which is bounding boxes you know and for those that are familiar with this with this world you know annotations it goes by a lot of different names of obtaining these labels you know this is is how you do cat dog mouse problem you know there that the world isn't labeled often with images so a lot of this isn't we're not the only ones trying to do this there are services out there that will do that for you but then again security comes into into place and some other kind of questions so we've we've gone at our ourselves going simple and what what the original plan was was to create templates and so we would we would take care to get one you know a few documents really correct and you can still get through these almost as quick as you can online sort of thing where you go with high speed but if we capture it in a certain way we could actually get real examples of that recorded data get that structured data and you know load a lot of different examples that it's not learning that the item of interest is a chair you know anything can pop in that chair blank or the dates we can rotate dates change the formats you know move them around a little bit but that's been the struggle is that we can't easily move around a formatted PDF or we haven't yet we're experimenting with that so far we have not done a great job so what they left us with is putting a lot of energy into getting a few templates that we could get a thousand of these out but a thousand different versions of one document even if we switch up the characters and all that kind of stuff we're overfit massively to the structure of these it learns that structure and if you put a new structure in it was - it stopped too early it can't it can't generalize to solve the new template and so that's kind of where we are at the more templates we add the better off it is but we started the batch size is down to 100 but really it could be as low as one so that's been one of the big lessons for me I still feel I like the power of having these templates where I can doctor them and work with them in a way that's not not typical with images you know the PDFs are structured a lot more than regular images I want to use that control and try to essentially randomize these perturb these just enough so that the deep learning doesn't over fit but that just enough we haven't gone we haven't gotten there yet so a simpler method is kind of what I learned Tudor earlier online annotation is is is a simpler way you know there's people to have this as a solved problem and their tools are just a little faster than ours but when you're doing a lot of these a little faster is helpful and so we've we've thrown you know thousand of these invoices into a tool where we're just kind of drawing boxes over things which is an interesting process to you know as I kind of said a few times we want different label types we have different models doing different things and so the one here I've got giant boxes of tables highlighted in the because one of the styles of models we want to look at this we don't want to worry about every single line item we want to have a model that is really doing kind of a simple first pass and so things you can do at a header level if you look at header in details so whether we had one line 100 lines one page 10 pages you know show us where the table of the data is you know that's just one model we want to use in our arsenal and so that's a different labeling state now if we wanted to use this same thing to get something else we're gonna have to click again and draw these boxes so it's been interesting labor-intensive is definitely something I think we knew we'd be up against because we had to go from zero to something and so now are there Mechanical Turk and other sort of ways of doing this yes there are and we certainly want our process is not automated well enough I think we would make the good use of that but we do we would look further because deep learning we just you need more and more data you know the better this will be and it needs to be of more variety we're missing in variety and we need to get there but that's where again we're three months in but we were paying it a lot of attention to this so our fourth member shanshan Wang is a data scientist aimed at this task she's our our training data owner really and so she's she's experimenting lots of ways of adjusting the PDFs to get so that we can get these deep learning models to work really well when we need them most which is when we kind of have crummy scans and all of our other methods have sort of aren't returning very good information the last bit is kind of how this plugs into driverless AI and we can do to two of these so that the part we're currently using driverless AI I think is a good story because it shows the power of driverless AI early on we were creating some of these template datasets you know we have lots lots of the data that would would fool an NLP algorithm not so much that would fool the image recognition the ones that need structure we have a hard time manipulating structure but the content we can manipulate a lot so those are more successful and those are the ones were paying attention to in a different kind of track if you will so Brannon sheket tend to be doing the deep learning models I've been doing some of these other ones and so one of the first ones to come alive was you know first started with almost first principles like let's just just pick the first date you see just you know as a benchmark how well does that do not very well but at least we have something so the next one was to use driverless and so realizing that a really simple way of what I'm looking for is to when we do have that embedded text or we can use OCR to obtain it which is the majority of the documents the large majority in fact of the documents you can obtain reasonably good text with that and that's kind of on the right if we look at every single line just in simple form I we can use our existing labels to figure out whether these targets were in there you know we once we have the bounding box as we had another way of going after them in simple terms and so the rightmost column is actually going to say the target class and this is just a standard machine learning problem at that point so with text you know with NLP so but we have that in driveless so so once I created this dataset fetter'd in the driverless and it's a again we're still struggling with the variance even for the NLP we've gotten a lot better since I've tried this model but it was it was at a hundred percent you know 99.9 AUC and almost immediately like within five minutes of running a driverless model the the work was to get the data to go into it then the model itself was it really simplistic because it's actually pretty easy to figure out all the terms we fooled it with you know we have a kind of a list of terms that we put in there to jostle a little bit and and it figured those out pretty quickly enough of it some of the structure a little bit of the structure it hasn't figured out too much vertical position on this one but it does a really good job of figuring out what we were looking for so the way of using driverless was actually pretty easy it's more of thinking of how you get the data into driverless and I think that that's kind of a data munging problem so here we're so familiar with our training process I think we will see this time and time again like there are just in the way that we're gonna have multiple different image models going after different variations of his targets I anticipate we will have multiple different driverless models out there while doing different things so right now the driverless model only take you halfway just like an image model will if it shows you the bounding box that's great we still need a formatted answer and structured data that's not the hard part necessarily the hardest part but we need it from here - because I've labeled the entire string you know it's a pretty stupid parser a simplistic I should say a parser every new line is is it and we just encoded as a target the presence of that string was there or not so we then have to extract the the data out of this one - but that's that's that's fairly simple actually with what we're looking for with two of these three targets very trivial in fact so this was nice to see I didn't have to think much you know that's the idea of driverless you know I spent all my time thinking of everything else with this problem the model itself I'd an access code at all I didn't have to do any of the typical NLP transforms you know you know those are done some of them are done we're making that better but you know we have all the way our target encoding and our text CNN was used for this one to do the job so that was pretty good the real goal of this project and this is kind of exciting you know a lot of a lot of the work we've done previously we haven't been able to include back into the tool other than you know reports so like it's helpful to have us with the customers you know before I was working directly with a customer you know as though you find something new every time you deal with the customer data said it feels like and that's true before I worked at h2o everything you know it's just kaggle I was amazed at the amount the diminishing returns you know just the law is just a little the slope is a little different than normal like it feels like you can keep doing competition after competition and just every dataset is gonna bring out something different it feels like and so here we have the opportunity to push this back in the drive I'll say hi and that that's connecting to you at some point our goal is so that you can pull down the results of our work you know sometimes that's maybe too narrow-minded for a lot of people and so you know a specific version would be there's invoice Tyvek something like that like will load a document maybe it's not an invoice maybe it's something else we have the terms we've looked at very much like a deep learning pre pre trained model in that world and a vector of what we think the probability is of the known classes that we've gone after in the past are trained on ours that's a possibility more generally you know also either in addition or if that proves too hard but I don't think it will that's really our goal is to get that in there in some way shape or form because we're gonna use it ourselves you know that's the best way of building these products we used to do build H 2 O 3 that way you know when customer didn't have what they need let's let's let's you know let's work on that so and that's the way we're kind of try to do this one but in general there's a couple other ways we can do that as well just taking the tips and tricks like that's how driverless was built it's taking grandmasters who've done you know hundreds and hundreds of datasets the learnings of that is how driverless came up from specifically Demetri but as it's been added on to so that's sort of what we could do to as we're experienced menteng with the image domain that we're not paying too much attention to otherwise h2o you know maybe we're allowed to to you know we're able to advise on how that comes into the tool or maybe we just extend the NLP models because we have that kind of need to we're already using the NLP models I can already see we need variants of that so we'll probably see maybe all three of these classes improve so and that's up we're out of time thanks for listening and hopefully you'll see their results of our work soon [Applause]