Send Varied Validated with airSlate SignNow

Send varied validated

okay thank you it's really great pleasure to be here and I just want to congratulate Gurley and the organizers for we did together a really great conference it's not coming to us really like the Jews are number two so I think we're people who join us in the bar in the evening by the mole here okay so my name is Mark I'm going to talk about a package set up to develop together with a colleague of mine Anthony okay today over here hold validate and I'd like to start with a small example to illustrate packages for so suppose I can't give over a father role and listen I guess in the audience and the angel over on the tele okay there's two columns it's a CSV file the first column is age person and the second column is income or celery so the minute I tell you it is gonna give you this violin you already have a couple of expectations right or simply you expect both columns to be numbers you probably expect H to be non-negative and maybe the same about celery and if you know something about labor market you might have some more complicated expectations for example persons below certain age ten fifteen years old have a salary plus would be zero and so these expectations couple the different variables that you're filing and the chances out it when you start analyzing a file like that and those expectations are violated then the results appear on that is which will also be incorrect all right this guy routine carries out so in general and this is only for you know expectations collect two variables but of course if you have many many many expectations and it's you know we would like to check them somehow so the purpose of the packets needs to make setting your data against such domain knowledge as an easy as possible so it's things between and so what I want to talk about in this talk is give you sort of a flavor of the package showing the basic concepts and workflow that we use really very simple I think and after that I just go through a number of examples showing the syntax so PSLV find so amusing arrange and typos and so this is the the basic concept we have three or four concept basically you start with rules so all these major patience you have about data you can express them as a short from age must be positive income must be non-negative if inches lower than 15 salary must be 0 things like that these are rules and we store them in a validator object the date that just comes to the data table frame at the moment we even from your data with the rules what you get is a bunch of results and they're stored in the validation object so we chose to store this in an object while I'm giving you a raw value to false or something like that because just like when you're building in your model says a lot of output at different it's just convenient to start and objects and the also functions that retrieve data that shows it for you so those are those are basically all concepts we have and this is how it works I will start the show me over a little bit data I'm going to using coming slides so in the battery packets there is a file included the dirty data included and we actually we have to include some new data because all the data example data and are so perfect it's maybe put some small example dataset in there and show me three columns here so I did these are data companies and then so we ask for turnover other income and total revenue and then just by looking at the first six rows you can see some strange things missing there are some revenue and one other thing you might expect is that's a turnover plus your direct you should add up the total revenue so there are lots of things that you can check okay so here's how to do it in validate one way to do it first you just store all the rules within the validation program you solace or just comma separated as many as you like you can confront that with your data and then there's all kind of analysis you can do on the call we are h2o summary which gives a short summary of everything that's validated so I have two rules well one rule - they're automatically named - v1 and v2 but you're actually allowed to define your own sixty items there's 60 records check 56 of them fast check zeroth and falen a couple of would not be checking process data missing so I'm checking whether turnover started at zero but if turnovers missing I cannot form a checks with anything better and actually there's an option you can also set that to false or true or sharing life by default to get a name that the check will be performed without errors or warnings and here's the expression that this user group over check yet so the second line is exactly the same you have some best process fails a lot of pennies it's not great because this one three parables or one na and the whole world can double-check but another thing I want to point out here is that we defined the quality rule here but to check that exactly on the murse is usually a really bad idea because if these numbers are computed somehow their bounty so machine around intimate sex advice equalities almost never the case so the fella data develop a package is able to recognize these kind of linear rules American rules and with some assumption of course and then okay another thing you can do is support so we have to see that there's a confrontation objective but it has only two rules on the x-axis you get the number of items read means okay that's yellow means to put in check it and rules yes Clark Walton you got it we know you love red so I think it's very easy to if you write markdown before she met the same date volume to check it like every week for example bottles like that so in a starting point despot okay if you want to expect every Brazil every rule every record and then as possible as well and the function values extracts the matrix just all angles now to emphasize native is that here I am only showing a rule so can you check record wise but it's not necessary okay here's a short hand if you think that defining rules and then using come from takes one line of code too much we also implemented this check net rule and so no bike operator is very popular nowadays so actually okay not a feature is everything I showed it allows you to do that oh come on line or less interactivity or maybe in a script that you write in our CEO of a script editor but it could be much nicer if you can store the rules in the file and then we use them and another really important that you use that is I have a file controls I can send it to the person who sends me the data and asking ok check it please before you send it to make sure all these rules so it's also a way to I think to communicate in the quality standards and other supplies so what you can do is just write those rules in a mini text file just like you with writing a script and then elevator you just read it a dog file are convincing that's it we have a second format and yellow format where you can add a lot of Methodism names descriptions things like that time spans so the rules can affirm that the neighbor by themselves maybe you want to that's your version or like the name of the person we invented to rule or make some kind of story by the rules you'd hold conditions so all that can be animated okay for the rest of the talk I'm going to just show you a bunch of examples of rules to show you the range of types of validations you to think of Indian so very simple technical checkers check whether a variable is Eric so any function that start with is Delta the ballots validation statements when we read in rules that you defined using validator or read it in profile all the rules can check and we check whether X means we'll evaluate it illogical all right so we're not any I was very flexible so we have stated check rather my fairy ball is down so supports that focal ref expresses it you can use the dot syntax and that was quite you should be introduces don't means the whole of the data frame itself and so the names document the column names or they'd set as a whole and check method dimensions there's a little bit more complicated who I was talking about suppose you're looking at me turnover percentage then you think you mean support divided by this check again song man living so what you're seeing is you can actually put any kind of are called calculation you've been nice you can do a linear model and check the size of them so meet the coefficient anything that's good to talk about what's important here says so that also we have amassed an element of syntax for checking conditional statements for example if turnover is zero total revenue must be equal to zero or in a company that was stuff the staff costs must be higher than zero for example so this is a normal arc statement but what the package does is it executes it for every record so we vectorize it and then the calibrated data sets it just makes it a little bit easier to write that each other an interesting case is this one you know called functional dependencies that's a concept that come from computer science or basically database theory and it's a very compact way of expressing that for example if two records have the same zip code and the city and the street name must be the same spell and this is sort of a mathematical construct this means two records have the same jetboat that the same city name and the same street name we've implemented it like this then there are a few things that programmability because if you have a lot of rules you you know you run into the chance that you're writing the same code many times so we've implemented sort of macro operator transit ferryboat variable that's defined and later all the way you can compute the median for example like this and then store it in a temporary variable called net okay and then use it in a rule so for large rule sets especially when they're in the day in a text file this may be really into these complications number of times another thing also kind of sure that the implementable groups so often we find that we have data sets with a lot of variables a lot of those variables need to satisfy similar rules so all of them have to be between zero and a hundred for example you have incentives so and our default case is put meaning for every variable you need to define rules margin and zeros or 910 so what you can do is define a variable group every group is in Department function and this equals operator and then these rules just get notified variables you can also combine with each other then you get Cartesian product okay because each animal I guess it will be the food service and then finally I think the referencing our data says so everything that pulls you up till now works if you provide other dates that a single data set and everything gets checked within the data center well there's a good probability that you have the same data center this year this week and next week you get a similar data set again you want to check over than things change like too much between them so you can actually call it and call them from and at reference data sets you can reference them so I think the syntax was designed to be really close to what you know from our we just try to analyze it and then there are several ways to do this you can use this environments so these are a bunch of features I didn't have time to talk about I will have four minutes okay then I also have to say that all these two function set here so have been implemented that we need to narrow creator or a sports in error so at the moment we have implemented everything for data frames as the data storage format but the plan is to support table objects have database connections and also dinner table in the future but we have to look at under the ancien version but everything is set up in such a way that we can actually okay so what I then shows you can really manipulate well they're all texture rules for us are really objects of computation so you can select them annotate them remove them and analyze them you can for example as from a valid data object this variables occur in all these rules so you can see okay these variables is some kind of check and maybe they also check at all you can summarize them I think this I mentioned also and there to forget this in the package one introductory and one that goes mini deep end all files because we try to put in a lot of propaganda programmability for example when you have a set of rules in text file you can have the text file include our two steps back in with and any other programming language actually go for the whole step so outlook so for us what they want to do is get more rule analysis more programmability for example GroupWise validation we have the manager the metadata specification let me now use yellow but would be very nice if you do it it's very close and the last thing I'd like to say is that for us validate Beckett is really the bottom of a stack of packages that we are developing for data cleaning because finding out that your data is not satisfying all you are desires is one thing but then you have to do something with it you'll probably like to fix it missing values need to be imputed by the way that rules must be fixed somehow so we have a whole bunch of packages some of them are from already are still under development for example the deductive package can rule set and see if better values can be derived from the rules as defined rules percentage profit kostas must equal turnover and i know for some reason that cost is wrong variables can you write should be and adductors package has a bit more fancy algorithm of course but they can do things like that that's one example if you're interested please have a look at our github data feeding organization thank you