Optimize your deal pipeline in Onboarding forms with airSlate SignNow

How to create outlook signature

specific steps that we needed to address detect that there's new data that is ready to be onboarded and figure out uh inspect the layout and format of that data then do the read of course the map the transform assess quality before finally loading it to the Target system and want to detect issues and log progress as we go through each of those steps but probably most important is that we want this sort of framework these seven steps to be a reusable process right a single generic data pipeline that will do these things uh for any data from any client right we lay these steps on their side and it already starts to maybe look like a data pipeline so now here's our first look at some actual Clover BX product right this is clover DX's uh designer IDE a as though someone was building a top level onboarding data job you can see the menu items along the top and various controls and panels around the edges all support the idea of developing in a low code highly visual way a onboarding data pipeline right let's zoom in and just look at the actual pipeline itself uh again the steps should start to become visible right detecting the data that's available here we're matching that data from a particular client with a configuration file before we do a read transform validate and push to the Target errors can occur at any point along the way you can see that most of these stages have secondary edges coming out of them uh that's where error information arrives we collect those errors uh builds a log entry and place that log entry someplace where our operators can uh you know react right successful uh ingestions get logged as well and then there's actually some post processing that's being done after the after the data is successfully onboarded uh you know you can call some other processor kick off some other Downstream process to further operate or or somehow act on the fact that new data has been injected uh into the system so now I mentioned in this process uh an interesting step which was uh matching a client's data with a specific configuration file right that's what's being done here in these two steps right data we detected data became available so we got some data from some clients we want to go look up a configuration file to give us instructions on how the rest of the pipeline should behave right so that's what's being done in these two phases so a word or two about configuration files uh because it's a very uh Key Practice for developing a single pipeline uh that's going to work for many different clients the idea of using a configuration files to hold all the client specific detail when you abstract that client detail out of the pipeline into some external config you're leaving behind essentially a generic orchestration framework right now the configuration itself can be stored anywhere flat files database tables uh we often like to use Excel for these configurations for several reasons they're they're human readable uh they actually conserve a dual purpose in addition to instructing the pipeline how to behave uh we can share this stock you know the the Excel sheet with the the end client say hey look this is how we're about ready to treat your data and if there are any issues you know that document can serve as a sort of a a launching point for discussions with a client on on you know how uh the ingest may have not uh worked as was expected right and of course uh excel's you know universally known is a lot easier uh to allow less technical staff right to build these config files to modify them into actually uh use them to operate the pipeline I have a real configuration file here we can take a quick look at again we're in the fintech space here looking to process Financial transactions config file that we did for this particular solution had three tabs you can see here across the bottom here overview attributes data quality first tab just some metadata about a client who's sending us this data uh what their name is a unique ID for them some contact information about uh who to reach out to if there are issues and some other metadata about expected arrival times uh for uh particular data right second tab has got the meat of the mapping right the records that the fields that we expect uh to be in the file that the client is providing us uh the data types that we're going to be treating them as some additional actual rules uh this column e over here is is specifying whether or not we should encrypt a particular field uh and land it in our Target system is encrypted so we don't expose any private information and maybe you know whether or not we should all as we're reading the file we should allow you know nulls to be in up here in the data set right so configuration specifically for this client's data set last tab on the configuration was some um data quality rules right so we can extract the client-specific data quality rules get them into a spreadsheet uh where the customer can understand them and then Clover can actually translate into runtime business rules it could be at the file level this file better you know have you know a certain number of Records in it uh a certain number of uh ID fields that are null particular Fields can have uh individual data quality assessments but all of this data quality rules for the client's specific data extracted out of the pipeline and placed into the spreadsheet so all of the information we need on board a particular client data set now in this file right the pipeline that uses this file uh when a new client file client data set arrives we match it with this config file and then the actual uh ingestion can begin right so ingestion right that's the next step highlighted here in the center the actual core of the processing reading the data and and um processing and getting it into our system right so let's uh let's dive through this particular segment of the of the top level Pipeline and look at at the underlying Clover job there right that looks like this sort of complicated so let's uh let's try to look at that look at it a piece at a time uh and point out some interesting uh parts right so first of all a Happy path right what we want is to follow the red line here which is you know to read the file do the Transformations do the quality assessments uh and then get it out to the Target output format right so that's the way we want all the data to flow through our system other pieces uh worth noting uh first you know the first step which is uh reading the file remember this is a generic pipeline that works on any client file right so with Clover DX we're going to allow you to design pipelines with reader components like this one that don't have any priori knowledge of the content that they're reading that information was extracted from the configuration file and used to configure this reader so it knows what it's supposed to be expecting in the file and that's happens uh at runtime this upper section here quick record count of the of the file uh make sure that we have roughly the number of records that we're expecting if not you know maybe reject the file obviously if it doesn't have any right if it's an empty file it only has like a a dozen records in it uh that's not a file that we want to introduce into the system in this case we were looking for a range say between a thousand and 1500 records is what you what we were expecting in the spot in this data set incoming data set uh anything that uh violates that we will simply stop the process that's a configuration parameter right a client specific that allows us to do that sort of data quality check transformation as I mentioned earlier uh this particular client had a key requirement that uh some of the data that was coming in from their clients needed to be encrypted before it landed in their systems so uh you know or in our systems right so this particular phase of the pipeline identifying which Fields needed to be encrypted again from the configuration file uh encrypting them and then reintegrating them with the rest of this record set before pushing down to the next step next up the quality right uh do the data quality checks make sure they're you know a particular date field that are not have any dates in the future uh this particular field better only have one of a set of allowed values that are you know specified in some look up it's gonna be a mix of uh of Baseline rules for your platform as well as maybe client specific roles for this particular uh data set right records that fail data quality are are logged along with a reason they're rejected uh the fact that we found in valid records at all is registered with uh the main job orchestration that's what's going on with this yellow component here uh but then more importantly we're creating a file a human readable file of the records that were rejected and why they were rejected those records can then be evaluated corrected uh and reprocessed so we have some ability now to rapidly re-execute our onboarding process identify those bad records fix them rerun the pipeline all without needing to have a developer step in right then of course the last step uh writing the transformed and validated records to our Target endpoint so uh when development of our whole pipeline of the top level orchestration as well as this detailed ingestion when that's complete uh the whole process is ready to be moved from Clover DX designer over to cloverdx server where it can operate in production right our cloverdex server will allow you to uh run these onboarding jobs automatically and unattended as well as monitoring those executions logging their results and alerting if there are any errors that occur so I'll pivot now and show you a couple of screenshots of cloverdex's server uh component and how it's handling you know all of the automation of this job that we've built right so let's talk about automat running these onboarding jobs automatically and unattended the most uh common way to do that is with our scheduler right so here's a screenshot of the cloverdex server console where we're looking at the schedule for and onboarding uh service right we can see graphically um when the last run was when the next run is supposed to be we can enable or disable the schedule or we can simply click on a button and ask the the process to kick off immediately without waiting until uh the next scheduled time so all that available to you in the console another interesting uh feature of the server uh well a design of clover in general is that all of our onboarding jobs the jobs that you create what we will automatically generate rest endpoints uh so that you can interact with those jobs on demand so instead of running on a schedule you can run when you need to right particularly useful uh in the onboarding process if you need to reprocess a certain data set or send through a set of corrected records here for example a little web form that sits on top of the API endpoint for our onboarding job we can specify a a set of records and maybe in addition to the set of Records we want to change the configuration file we would specify a new configuration file a couple of other parameters mash the button and we've got a non-technical user less technical user anyways ability to be able to affect when an onboarding job runs and the criteria that is being used to actually execute right see this a lot for uh error correction right bring in uh 5 000 records you know a few hundred of them fail we know what they did we know why they failed we fixed them we put them back at the front of the pipeline and it just runs again and uh this time just with the uh corrected record set right uh so lots of ways to automate uh or control the execution uh profile when our onboarding jobs are actually going to run we want to monitor the whole the whole system uh Clover has a dashboard that will allow you at a glance status of how your onboarding jobs are behaving right so if you see something red for example here that says you know the one of the onboarding jobs uh has come up with an issue with no at a glance if there's something going on we click this little card we can get immediately to uh detailed uh execution history uh for that particular instance of that particular job we know why the job failed we know when it failed we know where it failed uh because we have a graphical representation the same representation that we had in our design environment now is showing you how much data flowed through the pipeline before it failed and where in the pipeline the failure occurred making it very easy to triage and diagnose issues with your automated onboarding jobs