Save Heterogenous Text with airSlate SignNow

Save heterogenous text

[Music] welcome to the session on heterogeneous data and storage [Music] the advancements in computers and information technology makes the words changing drastically technology is the knowledge generated for the purpose of developing new systems to help in solving practical problems in the current scenario the impact of technology can be seen in all areas like home office and market data collection storage and analysis can easily be carried out with the help of technology with a huge amount of data collected we are able to analyze and predict the weather model space exploration and many more information is essential in all these applications since information gathering is easier in these days we are often facing with information overload and storage of information is crucial information processing is mainly done with computers and necessary hardware software services and supporting infrastructure is essential for the effective management and delivering of information the use of mobile phones to make calls and send text or multimedia messages the use of websites for ticket reservation and misenus use of ATM machines for banking and of course the social media requires enormous storage facility to store different kinds of data or information this leads to big data and cloud services heterogeneous storage large-scale storage systems are crucial components in the data intensive applications such as search engine clusters video on-demand service sensor networks and grid computing a storage server typically consists of a set of storage devices in such systems data layouts may need to be reconfigured over time for load balancing or in the event of system failure or upgrades it is critical to migrate data to their target locations as quickly as possible to obtain the best performance a storage node however can typically handle multiple transfers and this can reduce the total migration time significantly moreover storage devices tend to have heterogeneous capabilities as devices may be added over time due to the increase in storage demand the data replication mechanism of the storage system will have a great impact on system performance the key points involved in the placement of data replicas in heterogeneous storage systems include the determination of the number of data replication and the location of storage the number of replicas has great influence on the data availability of distributed storage systems few replicas could easily lead to overhead of partial replicas and our load of storage nodes if the number of replicas is large the storage resources will be wasted hydrogenous storage systems choosing those nodes with different performance and the different locations for replicas storage affects the quality of service when replicas are assessed the current application placement strategies can be divided into the source request placement strategy priority placement strategy path placement strategy nyebern odd placement strategy and random placement strategy etc lightweight adaptive replication is a typical source request placement strategy the advantage is that the new replication creation mechanism is triggered when the existing nodes reach the placement threshold which reduces the creation of redundant replicas this reduces the cost of replication creation and maintenance the disadvantage is that the load of the storage node is easily exceeded resulting in the unbalanced load of the overall system the advantage of priority placement is that when an access request arrives at the target storage node the other storage nodes will transfer the same replication to the visited node as a result the number of storage nodes is reduced the disadvantage is that the hotspot are easily general and the overall system load is unbalanced the path placement strategy is a simple and convenient method to carry all nodes on the request path when users accessed the replication but it is easy to cause the data redundancy which increases the waste of storage resources and the maintenance cost of replication consistency the nyebern on placement is mainly to save the history of the replication axis when a certain node is requested to reach the threshold it selects the neighbor node as the new storage node and makes the node access to the neighbor node the advantage of random placement is load balancing hence reducing access latency the shortcoming is that the number of replicas is too large heterogeneous data heterogeneous data are any data with high availability of data types and formats they are possibly ambiguous and low-quality due to missing values high data redundancy and untruthfulness it is difficult to integrate heterogeneous data to meet the business information demands there are different types of data heterogeneity such as syntactic heterogeneity occurs when two data sources are not expressed in the same language conceptual heterogeneity also known as semantic heterogeneity or logical mismatch the knots the differences in modeling the same domain of interest technological heterogeneity stands for variations in names when referring to the same entities from different data sources semiotic heterogeneity also known as pragmatic heterogeneity stands for different interpretation of entities by people data representation levels data representation can be described at four levels level one is diverse raw data with different types and form different sources level two is called a unified representation heterogeneous data needs to be unified also too much data can lead to high cognitive and data processing cost Slayer converts individual attributes into information in terms of what when where level three is aggregation spatial data can be naturally represented in the form of spatial grits with thematic attributes processing operators are segmentation and aggregation etc aggregation aids EC visualization and provides an intuitive theory level four is God situation detection and representation the situation at a location is characterized based on spatial temporal descriptors determined by using appropriate operators at level three spatio-temporal means that they should take care of both time and space related information a typical example is tracking of moving object which occupies a single position at a given time the final step in situation detection is a classification operation that uses domain knowledge to assign an appropriate class to each cell metadata metadata are crucial for future coding for relational tables and some extensible markup language that is XML documents explicit schema definitions in structured query language SQL XML schema definition existing or document type definition DTD can be directly obtained from sources and integrated into a meta model the XML technique is used for data translation the tricky part is semi structured data such as XML without excess D jason are partially structured Excel or CSV files which contain implicit schemas therefore the component structural metadata discovery that is SMD takes over the responsibility of discovering implicit metadata example entity types relationship types and constraints from semi structured data metadata management issues are important for an appropriate interpretation of heterogeneous Big Data detailed metadata are required some reports contain some metadata but many more details such as about the specific sensor used in data collection are needed for research purposes the collection of metadata and data provinces is a major challenge when the data are collected under stressful situations distributed systems the term distributed systems refers to and network the set of hardware or software components that communicate only by message passing unlike many other forms of parallel computing distributed systems will not have access to a shared memory the lack of shared memory allows them to be more scalable but introduces new problems with concurrency timing network failures and independent nod failures the term can be used for a large variety of appliances such as sensor networks the Bitcoin network and distributed databases due to the large amount of nodes in clusters single node failures occur very frequently GUI reported a mean time to failure that is MTTF of 4.3 months for servers in their data centers for a cluster with 10,000 nodes this translate to an average of one not failing per 20 minutes because of the high frequency of Nod failures the cluster must be fault tolerant to be practically useful data stored in the cluster should not get lost and processing should not be interrupted when a node or group of nodes filled this fault tolerance is built-in on a software level programs running in clusters must be able to detect and recover from a fault rioting fault tolerance distributed programs is very complex so many distributed programs use frameworks for error handling one of these framework is the Apache MapReduce model which is currently the main framework for big data processing the key features of a distributed system are components in the system are concurrent a distributed system allow resource-sharing including software by systems connected to the network at the same time there can be multiple components but they will generally be autonomous in nature a global cloak is not required in a distributed system the systems can be spread across different geographies compared to other network models there is greater fault tolerance in a distributed model price performance ratio is much better the key goals of a distributed system include transparency that is the ability of achieving the image of a single system image without concealing the details of the location axis migration concurrency failure relocation persistence and resources to the users openness refers to making the network easier to configure and modify reliability compared to a single system a distributed system should be highly capable of being secure consistent and have a high capability of masking errors this is referred by the term reliability performance compared to other models distributed models are expected to give a much funded boost to performance scalability distributed system should be scalable with respect to geography administration or size challenges for distributed systems include security is a big challenge in a distributed environment especially when using public networks for tolerance could be tough when the distributed model is built based on unreliable components coordination and resource sharing can be difficult if proper protocols or policies are not in place process knowledge should be put in place for the administrators and users of the distributed model types of distributed systems the nodes in the distributed systems can be arranged in the form of client-server systems or peer-to-peer systems client-server systems incline server systems the client requests a resource and a server provides that resource a server may serve multiple clients at the same time while a client is in contact with only one server both the client and the server usually communicate via a computer network and so they are a part of distributed systems peer-to-peer systems the peer-to-peer system contains nodes that are equal participants in that a sharing all the tasks are equally divided between all the nodes the nodes interact with each other as required as share resources this is done with the help of a network advantages of distributed systems some advantages of distributed systems are as follows all the nodes in the distributed system are connected to each other so nodes can easily share data with other nodes more nodes can easily be added to the distributed system that is it can be scaled as required failure of one knot does not lead to the failure of the Ender distributed systems other nodes can still communicate with each other resources like printers can be shared with multiple nodes rather than being restricted to just one there is advantages of distributed systems some disadvantages of distributed systems are as follows it is difficult to provide adequate security in distributed systems because the nodes as well as the connections need to be secured some messages and data can be lost in the network while moving from one node to another the database connected to the distributed systems is quite complicated and difficult to handle as compared to a single user system overloading may occur in the network if all the nodes of the distributed system try to send data at once heterogeneous distributed DBMS heterogeneous systems usually result when individual sites have implemented their own database and integration is considered at a later stage in a heterogeneous system translations are required to allow communication between different DB emesis to provide DBMS transparency users must be able to make requests in the language of the DBMS at their local site the system then has the task of locating the data and performing any necessary translation data may be required from another site that may have different Hardware different DBMS products or different hardware and different DBMS products if the hardware is different but the DBMS products are the same the translation is straightforward involving the change of codes and word lengths if the DBMS products are different the translation is complicated involving the mapping of the data structure in one data model to the equivalent data structures in another data model for example relations in the relational data model are mapped into records and sets in the network model it is also necessary to translate the query language is used for example SQL select statements are mapped to network find and get statements if both the hardware and software are different then these two types of translation are required this makes the processing extremely complex the typical solution used by some relational systems that are part of a heterogeneous DBMS is to use gateways which convert the language and model of each different DBMS into the language and model of the relational system types of heterogeneous distributed databases are federated the heterogeneous database systems are independent in nature and integrated together so that they function as a single database system unfed rated the database systems employ a central coordinating module through which the databases are accessed big data big data is a term used to describe datasets that are too big to process use traditional methods a study by Lani has identified three dimensions that force companies to adopt new architectural solutions and trade offs volume the amount of data has massively increased over the last decade ecommerce for example logs and processors all user transactions which leads to 10 times as much data as traditional sales the scale at which data is stored and processed has more from gigabytes to terabytes and terabytes to petabytes velocity the speed at which data is processed has increased and the expected response time has lowered companies move from periodic batch processing to near real-time or real-time processing for example moving from product recommendations in e-commerce with which has a relatively static product catalog to recommendations for new articles which change by minute or stock options which change every second variety the sources of data have become more varied in terms of data structure format and semantics these three aspects are commonly referred to as the three B's of Big Data other others have added more ease such as value which represents the increased financial gain companies have from their data porosity which represents the unreliability of big data sources and volatility which refers to how long that is valid and relevant because of these new challenges a demand has reason for systems that handle data in new ways examples of approaches are taking samples of the data to process only a fraction of all data investing in more powerful infrastructure and using different algorithms for data processing compared to traditional relational database management systems that rely on row based storage and expensive caching strategies these normal big data storage technologies offer better scalability at lower operational complexity and cost despite these advances that improve the performance scalability and usability of storage technologies there is still significant untapped potential for big data storage technologies both for using and further developing the technologies potential to transform society and business across sectors big data storage technologies are a key enabler for advanced analytics that have the potential to transform society and the way key business decisions are made this is of particular importance in traditionally non IT based sectors such as energy while these sectors face non technical issues such as the lack of skilled Big Data experts and regulatory barriers Novell data storage technologies have the potential to enable new value generating analytics in and across various industries sectors lack of standards is a major barrier the history of North QL is based on solving specific technological challenges which led to a range of different storage technologies the large range of choices coupled with the lack of standards for querying the data makes it harder to exchange data stores a Sigma tie application-specific code to certain storage location open scalability challenges in graph based data stores processing data based on graph data structures is beneficial in an increasing amount of applications it allows better capture of semantics and complex relationships with other pieces of information coming from a large variety of different data sources and has the potential to improve the overall value that can be generated by analyzing the data while graph databases are increasingly used for this purpose it remains hard to efficiently distribute graph based data structures across computing nodes privacy and security is lagging behind although there are several projects and solutions that address privacy and security the protection of individuals and securing their data lags behind the technological advances of data storage systems considerable research is required to better understand how data can be misused how it need to be protected and integrated in big data storage solutions it is time to conclude the session information provides the power to find and evaluate problems and make decisions effectively every aspects of life is getting changed due to the enhancement in technology which helps to collect more information and process this huge quantity of data very fast the storage requirements and the availability of stored information play an important role in this hence this module tries to find the storage of different kinds of data and techniques used to make them available as and when needed hope you enjoy the session see you next time until then bye [Music] you [Music]