Streamline Your Document Management Process with airSlate SignNow's Pipeline Integrity Data in IT Architecture Documentation

How to create outlook signature

data pipeline my name is Kenneth and I'll be walking you through this journey today I want us to cover storage so in a data pipeline there's many components in there the previous video we covered data generation today I want us to cover storage because storage is actually the Cornerstone of this whole life cycle because once data is generated even before it is ingested transformed it has to be stored or the data has to be generated then ingest to somewhere transformed and then stored so storage is very important and it's one of the biggest stages as you can see from the diagram here it actually is one of the biggest stages in a in a data pipeline so after data is generated where is it stored to be able to answer that question is we need to understand like data can be stored many times across a pipeline so for instance data can be generated from a source system let's say IO iot sensor data and then it can be stored right away and then used later for different purposes and then start again so data can be stored many times as it moves through the Cornerstone it can be stored and be needed to be used in within seconds data can be stored and used after a few minutes or it can even be stored to be used for years later so data must persist in storage until it's ready to be consumed for further processing or for transmission to a different spare to a different stage of the pipeline so when it comes to storage systems we're going to look at two aspects storage systems the first one is the storage systems themselves so the storage systems these are the raw ingredients that I used as a building blocks of these systems up here of the abstraction so the raw ingredients are what are the components that actually I used to build the big data storage and as you can see here the storage systems there's different storage systems like the Hadoop distributed file systems there is cash and memory based storage systems there's relational database Management Systems there is object storage systems and then the streaming storage systems together these systems can be designed to create the storage abstractions up here and storage abstractions in this new age of big data we have something known as a data Lake we have a data lake house we have a data warehouse and then we have a data platform which encompasses all these so we're gonna look into the storage systems and the storage obstructions we'll also look at the fundamentals that don't change within this storage pipeline so let's go ahead and start with looking at the Hadoop distributed file system so Hadoop distributed file system is a component of Hadoop Hadoop has two components there's the Hadoop distributed file system and there is a map reduce so Hadoop is no longer as popular as it was and right now we have different tools like Apache spark I have a video showing a quick overview of how we got from Hadoop to Apache spark but just to reiterate on a few key Concepts Hadoop was a big data habloop was a big data processing and storage system and it had two components it has a Hadoop distributed file system which you can think of like a catalog that's able to tell you where the data is and then there's another component called map reduce map reduce was what is used and is what is responsible for processing the Big Data so as I explained here is Hadoop dead when due to the Advent of this new technologies like Apache Spark this has shifted this as a spotlight from Hadoop because Spark provides more advantages over Hadoop because Hadoop was actually used mostly for batch processing but a patch a purchase pack can actually perform real time data processing so even though Hadoop is now the trendiest technology right now we still use its storage component which is Hadoop distributify system is still widely used in a lot of companies and it's actually a storage system if you want to learn more a little if you want to learn a little bit more about Hadoop to Apache spark and data bricks I have a video that talks about it in this in the next slide we're going to talk about cash and memory based storage so these are storage systems that are designed for Speed and this is stuff like redis memcache and what they do is that they store data in memory instead of storing data in the disk so this allows for low latency which means data can be accessed much quicker and they are normally used for like real-time data processing we have other common storage systems which are very popular and these are systems that you will interact with a lot and that is a relational database Management Systems these are just regular databases and they're used and designed to manage related data entities across different tables so we have different types of databases we have relational databases like MySQL we have postgres SQL there's Oracle databases there is SQL databases and one of the key advantages of relational database Management Systems is that they follow something called acid which is atomicity consistency isolation and durability these are properties are very important to ensure that data is reliable so that's one of the biggest advantages of relational database Management Systems if you go to the next storage systems we have something called object storage object storage is data storage is an architecture that actually became very popular with the cloud systems so what happens is in a object storage the data the architecture of the data and that manages the storage uh instead of having like other storage architectures like file systems and block storage you have a something known as the object storage so the data is stored as objects instead of blocks so ah these as I said it's actually very common in the cloud Technologies it's actually ideal for storing a lot of unstructured data and the examples of store of object storage is Amazon S3 there's Google Cloud Storage there is azure blob storage so these are examples of architectures that store data in the format of objects so another storage system here is streaming storage and these are designed to handle real-time data they capture and process large amounts of high velocity data from sources like iot devices logs and the two most common examples which one of them is very popular is Apache Kafka and then there's also Amazon kinesis so what this do is they're able to handle and store real-time data streams which is very popular nowadays because data has exploded and some applications will need real-time data processing capabilities and that's where Apache after Kafka comes into play and then now to go into storage abstractions these are the abstractions that use the underlying store and the underlying storage systems that we described so an abstraction is basically it hides the implementation of the of the data system so we have a data system here and it can use object storage you can use maybe the relational database management system but then the abstraction here is what um will actually use the underlying system to abstract and hide the implementation of the storage system so one of the obstructions here is something called Data platform basically a data platform is just an integrated technology solution that allows data located in like different disparate sources to be governed and accessed and delivered to users using an interface or a unified source of source of access so a data platform is the over aching system with various data Technologies for instance a data warehouse I did a warehouse is actually can be a part of a data platform so a data warehouse is a system that's used for reporting and data analysis integrating data from different sources I did a warehouse I can also be used to provide analytical processing and it enables complex queries and Analysis uh it's actually designed in a way that can understand and analyze relationship between data from different sources a very common data warehouse is something called snowflakes snowflake is very popular and is used in a lot of Enterprises as a data warehouse so that's a data warehouse it's very useful in today's Big Data Innovation and Big Data explosion we have another abstraction this is as you are well aware so data warehouse was very very popular and very useful but with every new innovation there's always challenges so data lake is what came into being it's a it's another like abstraction that was innovated after the data warehouse and what a data Lake does is it allows you to store a large amount and the repository of data in its native format so it actually allows both structured and semi-structured or structured or unstructured data and the reason why is in a data warehouse you have to store data that's structured so you have to sometimes you have to clean and process the data and store it in a data warehouse but with a data Lake it allows you to store both structured semi-structured or unstructured data so it's actually very important in this age of huge data explosion and then from there we have a lake house architecture a lake house is also new the difference between a lake house and the data lake is that a lake house combines a data warehouse and a data Lake into one so it actually it combines the best elements of what a data Lake offers with also the best elements of a data warehouse so it provides performance data management governance of data warehouse and then it's very low cost to achieve so databricks is one of the pioneers of the lake house architecture so this is something very new and it's becoming popular with companies because it allows you to be able to use a data warehouse and a data Lake in in one so instead of having to rely and clean and process your data and send it to a data warehouse and then have another data Lake for their unstructured or raw process data you can just use a data lake house architecture that will be able to do the two in one so this is something that's coming up and is very popular nowadays and then um right now we're going to look into the fundamentals of the storage system these fundamentals are what doesn't change regardless of what storage system you're using and one of the fundamentals is security is about protecting data from unauthorized access so in this stage in this storage system you have to ensure that the data is protected from breaches and this means that you have to implement some access controls because when you're creating a pipeline you have to interface with this storage system which means since you have access to this data you have to implement and protect that data from unauthorized access so you can set up policies that ensure that only authorized individuals can be able to access the data and then you can also set up some encryption to ensure that the data is encrypted the other aspect and the other fundamental in a storage system is data management so this one actually pertains to how the data is organized accessed and maintained in a storage system so for example in a system like Apache Spark it actually involves procedures such as data partitioning caching and persistence and in-memory management so with spark you could efficiently partition your data across the cluster to enhance computation speed so this is just like a an a procedure and an operation that can happen in a purchase pack to be able to efficiently process maybe a large amount of data so that's just an example furthermore Sparks memory management features will allow you to fine-tune and balance between storage and computational needs so what this says is that a fundamental aspect that you need to keep in consideration when you're dealing with the data storage system is you have to have a data management feature within it and then going forward we have to have data operations within the storage system so in this context of storage this just involves the principles and practice practices that improve the speed quality and reliability of data provisioning so this can actually mean implementing automated data testing monitoring Version Control for data and schema changes and continuous deployment for your data related code and configuration so if you're creating a pipeline that interfaces with the database with a storage system you can Implement these features to be able to have an efficient data operation data operations another feature is data architecture so this is just the overall design and structure of the data within the storage system so depending on the pipeline that you're creating you might want to select an appropriate type of storage so for example if you use the data lake house architecture and let's say using a streaming you're creating a streaming pipeline you have to choose the appropriate technology that appropriate storage technology that will actually hold the underlying data so in this case you have to consider stuff like scalability performance needs and just I should ensuring that the system will support the required data format and access pattern so this is something we'll have to keep in mind when you are creating a pipeline and you're within the data storage system and another component of the fundamentals within the storage system is orchestration so orchestration is just the Automation and the this is automating the management and coordination of data across different storage systems so you could be accessing different databases and for you to create an orchestration which means you have to implement some automation uh you you can be able to you'll have to Pro probably organize the order of how you will access this data so you can automate some tags tasks about the order of access you can also automate tasks such as migration so you can maybe migrate data from one database to another database so this is also orchestration so which means you have to keep into account orchestration when dealing with a storage system and then lastly software engineering within this fundamental of a storage system and what software engineering means is basically with the previous slide where you have to orchestrate when you orchestrating you might have to write some some code which means that you need some good data engineering practices and that means that so that your data can be maintained at a high integrity you can have good error handling recovery processes and just writing efficient data query so these are just some aspects that you have to keep in mind when you are dealing with the data storage aspect in a pipeline so um thank you in the next video we'll actually look into ingestion so we look into Data ingestion and the aspects and the features of a pipeline that you will have to implement in that stage thank you and looking forward to the next video