Sign Wyoming Work Order Easy

Document type sign work order wyoming easy

hello internet peoples my name is guy rice i'm a developer advocate at redis labs and today we're going to load csv data containing bigfoot sightings into redis uh we're going to do this not one way but two ways we're going to use riot and we're going to use node.js so stick around and watch how we do it [Music] with me guy royce let's go ahead and get to my desktop mode here and i can show you the two ways of doing this uh here is a nice up and running redis server here is uh information on riot we're going to do riot first and then here is a little environment for us to work in we'll take a look at our csv file first it's a classic csv file it's got columns on the top you know headers here so we got an observed we've got location details and county state title stuff like that and then it's got a bunch of csv values comma separated values representing this data and now what we're going to do is we're going to take the each row of this file and it's going to correspond to a single key in redis and that key is going to have be a hash that contains all of those values so row two here will be a key uh and in the observed value of that hash will be ed l with salmon fishing with the companion blah blah blah blah right and so we're gonna do that and then when we're done we'll have a whole bunch of keys full of hashes with all of our bigfoot data and we can use that later on for other fun projects so uh we need to get this into redis so let's look at the first way to do that and we're going to do that using riot riot is the way take a look here is uh the redis input output tool it's a command line tool it lets you extract uh data from redis and load data into redis into lots of different formats and from lots of different formats um it's an easy way to get csv files in here if we scroll down and look at the commands here we can see that this is the command for loading a bunch of beer data from a from a actually from a url into redis we don't want to load beer data we want bigfoot data not that i don't like beer but i also like bigfoot and so let's go ahead and i just copy pasta that over here because i'm lazy and i don't want to type it and we'll uh we'll pair it down now these settings will work because it's localhost and default but i'm just going to take those off and then uh so it's riot file import and i'll have a link to a riot file in the description below so when you go to do this you'll be easy be easy for you to figure out how to install this but it's just a little command line tool and we're going to take this geocoded csv file you don't have to tell it it's a csv it will figure it out by the extension and so we're saying it's a geocoded file it's a csv file and uh it has a header so use that for your column names uh we want to do an h set or an hm set in this case hm sets really deprecated but that's that's what we call it here this this triggers you to say this triggers right to say store all the rows and hashes as opposed to like say geosets or something like that um it's going to use a key space for that key name for each row you know we're going to have a key and it needs a key space we're going to do bfro citing let's just do let's do sightings and then it needs something to append to that to make that key unique uh it's saying use the column id this csv file does not have an id it does however have a number and i know the number is unique it's the um it's this it's the number you know it's like bigfoot sighting number three six seven nine uh saw bigfoot off of state route 33. that it's that sort of uh number but it's unique so we'll use that as our unique key and this is it i hit enter it tears through the file imports the file and it should be in redis let's let's take a look let's bring up a reddit cli we do a key star don't ever do key star in production kids it blocks the single thread for all the keys and if you have a billion that's a problem but for what we're doing here it's fine uh it shows 9172 sightings uh which is wrong i think i might have left over data from a previous run let me let me go do a flush db and then we'll exit out here we'll run it again real quick real quick and uh now we'll go in there and do keys star yeah i know that there are 4 586 bigfoot sightings so i had i i practiced this earlier can't you tell um so here we got our sightings uh interesting thing here there's sightings colon and then the number the number for whatever reason has a point zero at the end of it i think this is because the data set was used uh in machine learning and it was using pandas and pandas tends to wanna spit out numbers with decimal places on them so uh that's not ideal but let's let's go ahead and take a look at one of these uh sightings say h get all like that and we can see here's all our keys and values here's our sighting so someone's taking their dogs on a long walk with their girlfriends and they're hearing scary whistles and screams um these sightings are just so delightful to read i can't i can't express how much fun it is to take a look at them um but yeah there we go we we loaded them in it's it's a little funky uh it's not perfect it's not exactly what i want right our keys have uh you know our our number there is the id for the report and it has a zero in it i don't want that it's got some columns i don't want uh this has the weather data mixed in uh the idea here was that maybe the weather data might correlate to bigfoot sightings in some way like maybe if it wasn't very visible you would confuse a bear's bigfoot whereas if it was really clear then that would be less likely to happen but i don't care about any of that i just want to be able to build a little data set in redis that allows me to search for bigfoot sightings that are near me and so i want to filter some of them out the longitude and latitude here is really nice but i need to put them together into one column to use a reddish search to use them later on so um this is nice and quick and easy but this csv file doesn't do quite what i want and so i'm going to do this the other way instead of of loading this with riot i'm going to write some code in node.js to load this for me so let's do that so let's flush our database again and make sure that it worked it did we have a very empty redis database and uh i'm going to get out of the redis cli so i need to create a simple node project and that means i need to use the correct version of node i want to use the latest greatest and most updated version of node so if i do a node v i can see i'm using 15.8 that's great i want to make sure that anyone that's using node version manager always has the latest version they have the version that i built this with so you can create a nvm rc file and put the version number in there and then you can tell node node version manager to use that so if i say echo 15.8 to dot nvmrc and if i type echo correctly there we go and so now this file has the predictable value of 15.8 i can say nvmus and it guarantees that i'm using node 15.8 so if i'm like say i'm using node use default that changes my node version to 12.19 if i type node views switches it back so now i have the right version of node now that i have that let's create a project ultimately this will be the bigfoot data service where we're uh well we serve up information about bigfoot and so we're going to go and call it that version one is fine description uh helps us find the bigfoots the bug foots the buglets it's funny entry point we don't care uh test command we're not gonna do any tests for this thing yet uh git repository i might update that later but you're gonna get this code out of a git repository and so you'll kind of know what it is already keywords don't care author that's me put in my email address license let's do gpl 3.0 so that if you use this code you have to share it too that's all good now let's take a look at our package.json there is a big service version description main we don't care about main we're not building a node package to be included anywhere or imported anywhere uh we don't want to test script but it generated this one there's our author or license i want to do three things to modify this file i want to set them the type to module this will allow me to use es6 imports instead of commons.js or common jazz so i just think that's cleaner and nicer i just want to use that instead so uh type equals module and i want to make sure this is private just so that we don't accidentally publish it so that's the second change and then i want to add a script that will actually run my load code the load node code um so we'll say node load.js package.json is ready let's uh let's install some node packages we're going to use uh two uh node packages uh we're gonna use io redis and we're gonna use csv parser io redis is uh well it's a redis client uh written in typescript we're gonna use that it's it's the one i like to use and csv parser is um well it's exactly what you think it is it's a csv parser for node so there we go we've installed both of those things and now we can start working on this load.js file so let's go ahead and touch load.js bring it up in a text editor here and start importing some stuff so let's import csv from csv parser doesn't this uh you know these this import looks so much nicer than the require report redis from not reads i had a nickel for every time i made that typo from io redis i know that we're going to need to use the file system as well because we need to read that file off the file system and so there's our imports and so um we're going to need to connect to redis you know so so the idea is we'll use the csv parser tool to loop over the csv data and then add things to redis so i know i'm going to need a connection to redis so let's go ahead and do that let r equal new radius i also know that uh i'm going to be repeatedly adding things and i don't care about getting a response immediately and so this is a great candidate for pipelining so let's go ahead and create a pipeline uh pipeline and just allows us to keep feeding commands into redis without waiting for a response and since this is these are a bunch of unique rights that's a very safe thing to do especially since we're the only one messing with the database so we got a pipeline we've got redis now we can use fs to start reading in our file so we'll do create read stream and then our file name which is data slash bfro underscore reports underscore geocoded csv csv bfro reports geocoded csc yes i typed that correctly and so that's going to return a stream we're going to take that stream we're going to pipe it into the csv that we imported and so what csv is going to do is it's going to take this file and read the header and then read each row and it's going to generate an event called data that gets fired for each row in the file except the header and it's going to hand out a javascript object that contains um the you know just classic javascript object it's got the header is used for the property names in that javascript object and then the values are the from the rows in the csv file so if we do it like this we can just log everything before we pipeline anything at all um csv will also emit an event called end which tells us that we're out of data that we're done with data and here i would execute our pipeline i haven't really done anything yet but yeah once the pipeline does something we'll execute it and then we can disconnect from redis so this as it is should run just kind of show us that we're we're not insane to make sure that this is actually working so npm run load and we should just it should just start spitting out json objects with all the entries in our data file and that is indeed what we see we don't need to wait for that to finish but if we look here here's all those columns that were in that file so here's the observed location details the county the states uh monroe county ohio i've been there before um latitude launches this is all of our data so cool it's working it's reading the file it's spitting out results however we don't want to console log these we want to actually shove them into redis so that we can do things with them later so uh if we want to just shove these straight into redis unmolested we can simply say well we need to generate a key and we need them called h set so let's say let key equal citing we need to key space it right and prefix that with an id that would be data.number and then we can say r dot h sets key comma data save that and run that runs a lot faster because it doesn't have all this i o console i o to do and we look and read a cli and we look in our keys and we should see 4586 entries sure enough here we go if we copy one of those and say h get all we can see that we've loaded uh loaded all that stuff into redis from node which is pretty cool but now that we're in node we can actually start doing the transformations we want and so uh the first transformation i'm going to do is i'm going to reduce some of our fields so i'm going to use destructuring for that so we say what number i want the number i want the title of the bigfoot sighting i want the date that it occurred on i want the observed that's the best part it's the story the classification bigfoot sightings have three classifications class a i saw bigfoot class b i found evidence of bigfoot and class c someone told me they saw bigfoot class c is my favorite it's it's certainly the most believable um so we got our classification we want our county we want our state um we want our latitude we want our longitude and we want our location details make sure i didn't make any typos and spelling all those those all look pretty good sure if i made a typo you are seeing it and i will see it later and we'll just assign that from data so this will take all the values and data that match these uh these keys and set them to those variables and now we don't need to say data.number we can say number here and we can pass in a new object let's say let's values equal all those things there we go and now if we run this uh what will happen is has some values here we'll get just those columns so we've basically just pulled out the columns we care about assigning the local variables and created a new object with those same columns that's not the full fullness of everything we want to do right we want to transform some of these so let's do that let's transform first the id then the number right now the id is is it's right here right and 29441.0 i don't want that point zero so we're going to call parseint and pass a number to get rid of it so now we have an id we can pass in id and set a number and our key can be based off of id that's an improvement another thing that it does is the county has the word county after it every single time so if we look here we see if i can find the county here uh gila county i guess that's gila county uh we just want gila county's redundant we don't need that data so uh let's go ahead and say let's county now we can't use the light there so we'll just say county equals let's move that up county equals county dot replace and we'll just do a little regex look for the word county ends with the word county let's be very very explicit here place that with empty string so now county won't have that annoying problem anymore and uh the other thing we want to do is we want to take the latitude and longitude and we want to coalesce them or collapse them into a single value in in a way that um redis will like so we'll say let location and redis likes the uh latitude and longitude to be uh well not latitude and longitude but longitude and latitude so for our reddit search to do geosearch uh from values in hash we have to put it in longitude latitude order and have it separated by a comma so longitude and latitude and that's our location however we probably should be a little more clever here uh because longitude and latitude might be empty strings so let's say um uh longitude and latitude so if they both have values that are if they're both truthy then um you say let's let's lose eternity that'd be great launching latitude colon question mark if we have longitude and latitude do this otherwise do this so there's our id our county our location this should be all the transforms we need pass an id title date observed classi ication county state location and location details um it's bugging me slightly that this is not a let as well so i'll say let county um short county name there we go that's that's um we'll call it county name and then we'll pass that into here but we still want to put it in the county field so we'll do it like that we could actually just inline that but who wants to inline it come on man um cool so pulling out our data parsing the integer we're swapping out the county name to be a little shorter we're collapsing our location missing an equal sign there there's our key there's our values and then we call hsat and that should give us a beautiful transformation and loaded in redis let's let's try it out exit out of red a cli here that's not how we exit out of red of cli here we'll exit out of red cli and then actually let's go back in here real quick and do db or flush db just so that we have just the bits we care about make sure that there's nothing in there exit out npm run load there we go it runs go back into cli look at our keys again don't do that in production and if we do uh h get all citing let's go with 7940 so one of the things you notice here is that our id doesn't have that zero on it it's a lot shorter now here's our id here's our title here's our observed status ohio lawrence county ohio in 1980 there's our location longitude and then latitude uh classification is class a and here's details off county road 22 lawrence street road right near private drive 2050. so cool i'm tempted to uh write a little regex to get rid of that report 7940 because i feel like that's redundant so let's go ahead and do that too let's go ahead and say let title equal title dot replace and we'll look for anything that starts with report space and then we want um digits is it slash them i i'm being very brave here by doing regulars on the fly let's do report space dot star colon space slash that should match the beginning here um i can't remember what the the symbol is for is it not d i can't remember i think it's slash d so we'll do slash d star we'll see if that works we'll see we'll see if i can do reg x's here on the fly or not oops did the exit in the wrong place again uh that will let title better and then we'll replace title with title better here i think i'm going to do this um my my adherence my strong desire to uh have the let's line up is just not worth it here so let's let's do that exits actually we need to uh say flush bb now let's run that and then we'll run the redis cli let's do stu keys star oh wait key star won't work because it's time to wild card let's just go key star there we go um that looks better we'll say h get all and we'll just grab the last one there state county um undefined because i forgot to do this i replaced it with a missing argument and so it put undefined in there let's uh let's try this again flush db load go ahead and get that siding there we go nighttime siding by truck driver near red top mountain id is uh 19421 title is what i just read here's our observed as in georgia and bartow county and here's the date so cool that uh that worked so uh that's the other way so uh just sort of sum up um if you want to get csv data into redis uh and your csv is well formatted and well behaved and in a nice friendly format use riot if however you have csv data that is not exactly what you want it to be and you want to parse it and stuff this is a way you can do it obviously you could do this in other languages as well i'm using node because i like node so thanks a lot for hanging out um and uh watching uh me fumble through some of this code here my name is guy royce and um i'm just ahead in the corner but i'm here to help so if you have any questions feel free to hit me up on social media my twitter id is at guy royce and honestly you can just type guyros into google and you'll find me so thanks a lot and have a great one [Music] [Music] you