Enhance Your Quality Assurance Process with Data Conversion Lead for Quality Assurance

How to create outlook signature

so in this session we're going to look at data quality control so what is data quality control it's the process of inspecting detecting and resolving or cleaning data issues and why do we do it what are the benefits so if there are issues in our data we want to find them because this means that we can address them and fix them data quality is important no matter what type of data you're working with removing errors and inconsistencies increases reliability accuracy and credibility of the data and that's what we want to achieve we want our data to be high quality we want quality evidence that will answer our questions so looking at the data quality control workflow it can take many forms but generally a good process to follow is starting with inspecting the data trying to uncover any problems and then cleaning the data verifying that the data has been cleaned correctly um if it hasn't we can go back and inspect again clean and when we can verify um we document the data quality control and that cycle can continue on so there's no one fixed way to carry out data quality control the methods that can be used for inspecting and cleaning data really depend on the type of data the amount of data who's using the data but we're going to look at some of the more common data issues that often occur in data sets so we're going to look at completeness location data duplicates incorrect observations data types missing values data levels units of measurement categorical data text formatting and contradicting values so a good place to start with data quality control is to look at the completeness of the data is everything included that should be there do does the data have the expected number of data records you can check the total number of observations in your data set you might know exactly how many observations there should be if it's not what's expected maybe you don't have the final version of your data maybe you have a version that was taken during data collection and there could actually be more data somewhere else you can look at the number of columns and look for any gaps in the data so in this example data is being collected at four markets and by looking at the count of the surveys we can see that we don't have any data for market number two and if we know how many surveys there should be from each of the markets we can also tell if we have complete data sometimes it's really helpful to visualize the data so in this example we have met data from four different med stations and this graph is showing where we have data for the max temperature and where we don't the gray parts are showing where the data is missing so looking at station number one um maybe we already know that during 2020 there was a problem and that did it wasn't recorded so that's fine we know that we don't have that data but maybe by looking at this graph we also see that there's data missing in 2019 um and perhaps there shouldn't be so maybe the files for that data haven't yet been added into the data set so that could then be resolved if your data contains location data like gps coordinates actually mapping that can reveal issues so even if you're not using the coordinates in your analysis um it's very helpful to plot them in this example here we have a survey that was to take place in rural areas we can see um in the bottom right corner we have data points lots of data points down here then we also have seven data points over here in this urban area and so what's actually happened here is that data was collected during training maybe somebody was testing in the office and that data hasn't been removed from the final data set yet in this example here we see that the importance of having longitude and latitude correctly labeled um these coordinates on the left show us a location off the coast of spain middle of the sea and if they are switched we get a location in kenya so that's something else that's important to look out for a duplicate record is where the same piece of data exists in the data set more than once duplicates can be caused by input error so maybe whoever collected the data or entered the data it didn't know that it already existed or maybe the duplicates happened during merging of data and duplicates should be removed incorrect observations should be removed or fixed for example if an age was recorded as 160 years should that be 16 years or should it be removed um and outliers as well they're not incorrect observations they are real observations shouldn't necessarily be removed but they should be verified it's important to check the data types so that the data can be properly used during analysis um they can be checked by running summaries of the data and if they're not correct they need to be converted to the correct type and here we have some examples of data types numeric we have dates categorical boolean and text missing values should be handled carefully and they should not be ignored um and it's important to understand why the value is missing to be able to take the correct course of action um another issue is having different levels of data so you can see here we could have data at the level of the household we could have um many different members of that household we could have did at plot level and then we could have multiple crops are plot so the key here is making sure that the data is correctly linked together using ids as this can cause lots of issues units of measurement should be standardized so you can have different units for currencies area distance and time could be measured in days and months so we want to standardize these units for example here we have weight and it has been recorded in kilograms and pounds so to be able to use this data it should be converted to a single measurement unit and you can see here that we have coffee harvested and the original units are listed there and we've added a column in converting the harvest into kilograms so categorical data um the text formatting should be standardized without spellings should have consistent capitalization and no white space at the beginning or end here's an example of a categorical variable and here we would actually have seven categories instead of two because these values are all different because they're formatted differently um so by fixing the spelling um we get this it's looking a bit better um using consistent capitalization um doesn't really matter which you use if you use lowercase uppercase capitalize as long as it's consistent and then by removing the y space at the start or end we get this so contradict and values can occur within the same record of data or across data sets that are linked together for example in a data set containing the costs of producing a crop the total column should equal the sum of the individual columns so you might have a column for the costs of seed labor land and so on so it's possible that there could be errors they need to be fixed and moving on now to documentation um it's really important to keep a log of data quality issues and to note any data cleaning that has been undertaken um always keep a copy of raw data files and save process data as new files using appropriate file naming conventions um it's good to keep track of these changes because errors could also occur during data cleaning and if you've kept a good log and kept all the original files and those errors can be resolved as well and then metadata should also include record of any data that has been removed from the data set or modified it should include any limitations of the data and a description of the level of the data quality