Optimize your Engineering Pipeline Management with airSlate SignNow

How to create outlook signature

hey everyone shashank this side currently working as a senior data engineer at expedia and previously work for companies like amazon so in today's video i'll be talking about the process of designing a data pipeline so before starting our actual discussion related to data pipelines if you are new to our channel then make sure to subscribe and press the bell icon so being a data engineer our actual road and responsibility is to design optimized scalable and fault tolerant data pipelines and based on different business use cases sometimes you might be working on different different data domains as well let's say you are working for an e-commerce company then data would be related to the e-commerce let's say you are in the telecom so you will be calling it as a telecom data domain similarly healthcare finance banking and multiple other data domains so the very first step of designing the data pipelines is to understand these data domains in a correct manner and this is a really really crucial step when we design the data pipelines we invest our most of the time in this phase because understanding of the data domain is really really important in your organization or let's say you are working for a client which belongs to a specific business they might come to you with a specific problem statement and being a data engineer if you don't know about that particular data domain you won't be even able to understand the important terminologies and like what kind of matrices they are expecting from this kind of data so getting that understanding is important plus this will also help you to identify what would be the multiple data sets you need to consume from the sources what would be the important relationship among them like the connection with the help of primary keys and foreign keys and this will eventually help you to identify that what kind of columns or what kind of important data sets you need to bring into the system because the end users or the clients sometimes will come up with a big problem statement and they would expect you to bring every single data set and being a data engineer you always look forward for the optimization scalability and fault tolerance and it's not recommended or it's definitely a bad practice to bring the data sets which is definitely not necessary and if you don't understand the data domain the relationship and how these matrices are getting generated then in that case like without any knowledge you are just bringing everything and you are adding extra cost in your project because it will be unnecessary data processing so this is the very first step understand your data really well and then the other technical steps will be involved while designing the data pipeline so the second most important point while designing the data pipeline is the choice of the data sources so while consuming the data or the data extraction process data can come from multiple sources sometimes the databases distributed file system and apis and i'll tell you why it is important to understand the data sources so as we know like data can come from multiple sources based on the type or based on the let's say olap oltp data or api related data so we can consume data from multiple sources and that will be completely depend on the businesses let's say they are using the mysql database for their data storage so you will be consuming the transactional data from there and let's say you are working on a use case where you need to process the data which is coming from sensors then you will be getting it from iot devices and let's say there is a web application which is generating some data for you so you can consume it via apis as well so understanding the core capabilities of every data source is also very important because that will help you to understand which kind of computation engine you need in between which can easily connect with those data sources and pull the data from there so if you don't understand what kind of data source you are going to use probably you won't be able to come up with the efficient design and also the important part is you need to understand how this data is getting populated on the source side as well whether it is a real-time data or a batch data or how it is getting stored there the file formats right these kind of things you need to understand from the data source perspective and that would be the initial step of your data extraction so the third important step in the data pipeline design is to determine the data ingestion strategy so in the data processing there are majorly three important data engagement strategies first is incremental data load second is the full load third is upset kind of load and i will explain each of them one by one so first let's talk about the incremental data load and when i am talking about these data injection strategies so it will again completely depend on your business use cases and how you are going to perform the analytical queries so first let's talk about the incremental data load so in that case what will happen whatever data source you have picked so based on some time stamp values you will try to bring the data in smaller chunks let's say first time you just hit your data source at let's say 11 pm and till that point whatever data was available in the data source you pulled it and you dumped it into somewhere after the processing and after let's say 15 minute your pipeline again executed so from 11 pm till 11 15 pm whatever updated or newly created data was present there you just pulled it and processed it and then again dumped it into the downstream system so this is the incremental load like based on different different time stamps you are pulling the data and dumping it into the downstream system now talking about the second data injection strategy which is full load so in full load it will mostly depend on the like use case if every day you want to process something so in that case what will happen like whatever data is available at your source end so completely like you will just erase your target table which was previously loaded with some data and whatever fresh data is available for current day you will pull that entire data set will process it and dump it into the downstream system and the third strategy is absurd kind of data ingestion so upset kind of data injection what will happen this is also uh just a variation of incremental data load so here what will happen let's say you pull some data records and among them few records were already present in the target table because you process these kind of records few days back and that relation can be checked based on some primary keys so if you find these kind of common records what will happen you will remove those old entries from the target table and whatever new record you have pulled you will ingest it so that is a kind of updation we are doing or not let's say we are not deleting the records from the target table we can update few values of it and the two data injection strategies the incremental and absurd kind of load that is mainly designed for the historical data as well because every day you are processing everything and you will be keeping it for some time this time let's say till one year and two year you will be processing these incremental data and this kind of injection strategy is needed when the analytics team want to analyze data per day basis per hour per week something kind of this strategy then you need to like the dump data into this form like incremental or upside kind of load but let's say your analytics team is only concerned about for specific range or the batch data let's say one year or one day one month six months so in that case you will just pull the data for that entire time window which the analytics team has provided and will do the full data load so that's how you can understand based on the use case which data engagement strategy you need to pick for your data pipeline so after completing these three steps the fourth step will be to design the plan for the data processing so so far we know in the big data domain we have multiple frameworks available for different kind of data processing so the important data frameworks which are available hadoop is there spark is there flink is there so based on the use case like you want to do the batch processing and real-time processing you need to decide which one is a good fit and i will explain how you can decide it so in this data design plan or you can say data pipeline design plan you will majorly focus on the choice of the data framework so let's say you are concerned about the batch data processing only then you should pick the kind of framework which is a right fit for it for an example there are two popular frameworks like the hadoop which is completely designed for the batch processing and also you can leverage the spark as well which is the most demanded skill set for the data engineers as well let's say you are designing a near real-time data pipeline then you can leverage the spark streaming and let's say you are trying to create a completely real-time data pipeline then you can leverage the framework like flink and along with the apache kafka so based on the kind of data processing you are trying to do you need to pick these kind of frameworks and the second thing which you need to consider while designing the data plan or while designing the execution plan for these pipelines that it should be completely scalable completely optimized and fault tolerance and how you can achieve it if you know the exact comparison between all the big data frameworks like hadoop spark flink so based on the criticality of your data pipelines you can pick whether this particular framework will help me to achieve the scalability part default tolerance part and the optimization part and this is the most crucial plan and don't take it lightly because your architecture or your data pipeline ingestion will completely depend on this one because these are the frameworks which will help you for the data transformation so try to get the good understanding of the main and core capabilities of each big data framework so after the data processing plan you would have coded all the logics like whatever business rules your business teams has provided to you you would have coded it like on the source data sets whatever transformation deduplication logics filtration data validations multiple things you would have done and the next important step would be to decide the downstream system or the storage part for the transform data and this is also one of the important point to be considered in the data pipeline design because based on the use cases you will decide which data storage is the good fit for your use case so i'll give you the examples that what kind of data storage you can pick so so far we have multiple data storage available as well let's say you want to process or transform the data and dump into the downstream system like the s3 and any persistent system like hdfs and we will be also having options for the nosql databases so let's say your team or the analytical team wants to query data time to time very frequently and ad hoc queries right so you will be looking for a kind of data storage which is uh scalable and partition tolerant as well and here the nosql databases like cassandra mongodb elasticsearch will come into the picture because they are the good fit for the analytical queries and let's say you are trying to dump your transform data either in the data lakes or in the data warehouse so based on these two terminologies like data lake or data warehouse you will decide which one needs to be pick so let's see if you are trying to create the data links you can choose the persistent storages like s3 hdfs or anything like that and similarly for the data warehouses you can either pick any cloud service like the aws red chip or any other open source data warehousing service like apache hive so that's how you can decide based on the business use case or the end analytical impact on the queries and how they gonna consume it designed it and let's say you are processing the real-time data and somehow your dashboards are getting populated with the real-time data so you might be also dumping data into the transactional databases like mysql and postgres so based on the like batch or real-time data processing or the impact on the analytical queries you can decide the data storage so after deciding the data storage part the next step would be like scheduling your jobs or the workflow management so in the big data so far we have most efficient workflow managers like apache airflow azkaban and apache nifi so based on the use cases you need to decide that how you gonna schedule your pipeline and how do you wanna create the dependencies so i'll explain it with an example let's say you want to derive or you want to populate a table which is named as stable c and this will be derived with the help of two data sets let's say which is stored in the table a and another data set is stored in the table b and we have even created a separate pipelines for the table a and table b and based on some transformation logics we will be populating the table c so here we can see this uh interconnection or the dependencies like the pipeline c or the tables is completely depend on these two pipelines table a and b so we should definitely have some mechanism in place which can take care of these dependencies and also can help us with the schedule part as well because multiple times we might be running our pipelines on specific times let's say one hour five minutes ten minutes so these schedulers or workflow managers will help us with the pipeline dependency management now the last step of the data pipeline design is to set up the pipeline monitoring or the governance tools so this is also very much important because you cannot assume that 24 into 7 your pipeline will be running successfully there can be multiple instances where your pipeline might get failed because of any unavoidable circumstances so what you need to do you need to set up some mechanism so that you can monitor the health of your pipeline or what kind of incoming data you are getting the scale or the peak volume so that you can identify that how my pipeline is working for a specific time stamp and you will be also able to like log the all kind of errors or any kind of failure if it happens and this step is really really crucial because sometimes there are pipelines which are really critical let's take an example of financial data right which is really really important for any bank even to figure out if there is any fraud transaction is happening and even for a one minute or two minute your pipeline is down and you are not able to get that alert or any kind of notification that means uh you will be losing the customer trust and that's why these kind of monitoring tools and governance tools will come into the picture and we need to use it for all kind of monitoring failures errors the health and these kind of tools like we have graffana we have datadog we have the pager duties which will help us for these kind of pipeline monitoring now i will take an example of dummy created pipeline and will try to explain you that how we design it with the help of all those points or the steps which we talked previously so here is the pipeline and one by one i will explain whatever points we talked about how this pipeline is taking care of it and how you can even design these kind of pipelines so as discussed first thing let's say you got all the understanding related to the data domain you understood it and now the second point was to choose the data source so here what we are going to do we are trying to pull the data from the transitional database like mysql okay so this will be our database where let's say some web application was there which was dumping the data into the transactional database and our analytical team wants to just consume this data with some transformation and they want to analyze it so now being a data engineer it will be our responsibility to create this pipeline end to end along with the transformations extraction and the load part right so the database part is here now the next part was the data ingestion strategy so after deciding this mysql part so we know this database will have multiple tables and those tables can have timestamp columns as well so we can use those ones in order to get the incremental data so here the strategy we are going to follow is the incremental load after that there will be the data processing plan or you can say the data processing framework which will be doing the actual transformation so here what we are going to do for the incremental load we are using the spark so apache spark will be the framework which will consume this data from the mysql database on increment basis and will apply all the business logics uh the duplication logic the filtration the data validation and let's say there was some join operation group by operation whatever is actually needed to create the transform data for the business analytics that will be done and taken care by the apache spark right so that was about the data processing framework part now after that the next step was this data storage so here again based on the use cases like whether we are concerned about the data lakes or the data warehouses so data links are mostly for the code data or there we are trying to dump the data in its raw format so that it can be used for multiple use cases and mostly for the analytical queries for very long period of time let's say one month of data and one year of data or even a one week of data so that kind of queries this uh data lake part would be efficient and we will be storing our data there so for that one we can use either hdfs and also aws s3 if we are going with the cloud or let's talk about the data warehouses because data warehouses are the services where we dump our transform data actual transform data which we have prepared in this spark application that will be done into the data warehouses and let's say we are generating four or five transform data sets so we need to generate it in such a way so that there is a proper relationship among them so that we can use it with the help of the join operation to get the meaningful insights so data warehousing services if we want to use we can use apache hive as well and if we are going with the cloud services then aws redshift is a very very popular data warehousing service now this part is designed our source was decided our data processing engine was decided and then our output storage was decided next point was to schedule this pipeline and since we talked about the incremental load so let's say what we are using we are using apache airflow here and here we are scheduling our pipeline to get executed after every 15 minutes right so apache airflow will take care of this triggering point and will keep on monitoring the time stamp for every 15 minute of window it will trigger this pipeline and this pipeline will do its job for the incremental load and whatever data we are processing so we need to monitor it as well like the data governance part which is very very much important so we can just transfer the logs whatever we are generating or whatever we are doing in our spark application that can be transferred to some logging system and on top of that we can use these monitoring and governance tools like the data dog and pager duties so if there is any kind of mishappening which is currently going on with our pipelines that will be captured by these government tools and they will notify us and we can quickly take the actions and at the end like the last layer will be the presentation layer where either we can populate any dashboard which will be used by our uh business users for the query and the data representation part or let's see we can provide our direct access to our data warehousing services where our end users can run their ad hoc and analytical queries and they can populate or you can say they can get the meaningful insights of this transform data so that was this entire architecture and i have covered all these steps and this is the most generic way to design any scalable fault tolerant and optimized data pipeline so that's what i had for you guys in this video and i'm pretty sure if you are an aspiring data engineer then you would definitely take this video seriously and we'll be following all the important steps which i talked about for the data pipeline design and if you find this video really informative make sure to like it and for more such type of content subscribe to scalar