Send Myriad Placeholder with airSlate SignNow

Send myriad placeholder

hello everyone my name is mohit i worked for paypal and i have here santhosh with me he works for me Power Technologies and today we are going to talk about resource sharing Beyond Boundaries using apache myriad so before we get started like I want to know how many people have heard about myriad before coming to this talk like a show of hands nice so when how many people use yarn for work or fun okay and how many people like want to use yarn and mrs. together oh nice okay so how was it mrs. Khan last year and I gave a talk on what was precursor to a myriad and it's been over like one year and the project has come a long way like we have about like 200 something stars on github and be caught in Cuba tat under Apache earlier this March and a lot of good work has been done over one year and we will try to cover most of that during today's talk and before we can get started I would like to thank these individuals without whom mediate would not have been where it is today so a brief round of applause for all of them so so we're going to talk about what's up with data centers these days and then how may sauce and yarn compared to each other and why would you want to run both in your data centers and how myriad can help you with that so what's running on your data center well you have some tier 1 services these are like some business critical services for example your checkout service or maybe your login service or maybe the service which takes the kappa reservations and these services are very critical for the business because when they go down they really affect your business directly and then you have some tea or two services a good example of these can be your build system or let's say your QA environment and they are not as critical str1 but they are like really helpful for your development environment so if even if they go down I think it's it's fine for a while and then you might have some high-priority bad jobs running a good example can be your billing match which runs every night or some other data analytics jobs which should run when they are supposed to run so still very high priority and then you have some other best effort backfill jobs that can just run whenever somebody wants them to run but they're not really very critical and as we can see these are a variety of workload right and then if you want to run these on your data center you want to have a resource manager with like following requirements you want your resource manager to have a programming model that is based upon resources and not machines and adding anyone who have seen scripts and programs with hard coded IP addresses can feel the pain when when you think about machines and not resources and then you want your resource manager to support custom resource types like for instance you might have some specialized hardware like GP GPUs or you might want to take into consideration the power consumption of the CPUs while scheduling etc and as we saw in the last slide we have a variety of workload right so you might want to have customized scheduling algorithms for them for example you might have some jobs which come in at a very high rate and then they run for a very short interval of time so you want to like schedule them as fast as you can compared to let's say a database that you want to run or a caching server that you know that is going to run for a couple of weeks at least so you when you are scheduling that you want to be extra careful like on which machine you are scheduling that you also want to have lightweight executives that can launch your task really really quickly without adding extra overhead and you want to run all of these workloads on any node in your data center right you don't want to create silos so you want to also support multi-tenancy and to support multi-tenancy you need to have a support for strong isolation and with with that you can like drive up your utilization and you also want to support the entire big data ecosystem and Hadoop together on the resource manager right and you also don't want to leave behind your legacy systems so you want to also schedule them dynamically as the newer applications that you are building and a strong support for containers is also a much there and lastly you want to connect the big data to your non Hadoop apps because they cannot no longer live in like silos for it for instance you might have a front end surface generating lot of data which goes through your analytics pipe in and then you want to like feed that back into your application so that they can use that so the HDFS and the non Hadoop apps like cannot live like they have to like lift together right so to like solve all these problems we believe that may source is the right buster manager so this is my sauce con and I'm pretty sure you all know what my sources by this time but it's still it's an open source project Apache project sequester resource manager scalable up to tens of thousands of nodes and its fault tolerant it doesn't have a single point of failure has really good support for multi-tenancy and resource isolation and lot of people who have been using these hoes for a while they have reported improved resource utilization and missus is more than yet another resource negotiated or yarn I mean it has great support for long-running services and real time jobs and it has native support for darker and the sea group support in masers is there for years and it's more complete than yarn it can support CPU memory does networking etc and it's really easy to write a framework on top of me source like you can write a framework like a toy framework in at least 200 lines of code and the core of mrs. is written in C++ of for performance but you can write your frameworks and executives in any language that you choose and like these are few companies which have been using mesos for a while and I think there are many more now and I was not able to put them all on on this slide so so this is a brief overview of like mesos before we can go further so a maze host cluster has a pack of masters which was led by one active meso Smasher elected through a zookeeper quorum and these means host masters are aware of all the resources that are in the cluster and there's another component called me source agent like denoted by me so slave in the slide I haven't corrected that yet so these mesas agents are basically report how much resources each node have in the cluster back with the master and they also assist master to schedule any tasks on the node and then you have the frameworks which also have a scheduler in them so may source is the two-level scheduler and mrs. masters can do resource scheduling whereas the frameworks to the task level scheduling now let's compare that to yarn with analogies right so in yarn you have something called a resource manager which is very similar to a mace hostmaster it is aware of all the resources in the cluster and can like schedule tasks and then you have node manager which is very similar to am a source agent it reports back these horses to the resource manager and assist resource manager to launch tasks and then you also have something called an appt master which is kind of similar to a masseuse framework but not exactly the key difference being the app master only expresses an intend to launch tasks but doesn't really do any scheduling so all the scheduling is done by the resource manager itself so it's sort of like a monolithic scheduler and now the question is why do you want to run both of these together in your data center and I think the short answer is both of these I don't see them as resource managers I see them as ecosystems and both of these are like really strong ecosystems may source being a general-purpose resource manager can schedule like I can help you schedule any type of workload and it has a really good support for long-running service whereas yarn still has a lot of big data-related frameworks and I'm an applications written on top of it so in a big enterprise you really want to use both of them together but what happens when you try to use them together well in this example on the left you have the green nodes like which are managed by am a source and only missiles can schedule tasks on it and then on the right you have the blue nodes and only yarn can run any task on them and the dash 2 rectangle that you can see in this slide like the note said there is a inherent static partition in your data center and like honestly like static partitioning like sucks because well your Hadoop teams might be fine with the isolated clusters but if you talk to your Ops guys they are not they are really unhappy with that because if node goes down in your let's say Hadoop cluster it's like really it's a very slow process to bring back and provision a new node and like iron back to the cluster also like these the static partition also creates a resource silo which reduces the elasticity elasticity that you have in the data center for instance let's say you were running your frontier on main source and there was a traffic spike and you want to offload some of the extra work on to the Hadoop cluster by shutting down some jobs which you cannot really do if there is a static partition or the vice versa right like at night you have less traffic and you want to run like lot of lot more Hadoop jobs you cannot borrow resources from the maze of semester so like that is bad and you also like we talked about it earlier you also want to run Hadoop on the same infrastructure as that you're as your tier 1 services without really interrupting them and to do that you also want to support like multi-tenancy and resource isolation and ideally if you want to go to a model where both masers and yarn can co-exist happily in the data center and you don't really distinguish between the nodes in your cluster so mesas and yarn can schedule on any node in the data center so in this example you have you can see that there are certain nodes where the green task which is the mezzos task and the blue box which denotes a yarn task are running together and this gives you the elasticity and the flexibility to schedule any type of workout on a data center and now I will hand it over to Santosh who is going to talk more about media and how mediate can help you achieve that so take it away the notion thanks moe and thanks everyone for coming for this talk so so let's actually go further to see how myriad can help in solving the static partitioning problem so just a quick worry about myriad myriad is a full-fledged framework for running yarn on top of missiles so missiles manages the whole data center and media tries to run yon as a framework on top of missiles and media tries to negotiate resources between yarn and missiles and in this talk we are going to actually further look at the resource sharing models we have a couple of resource sharing models which we feel will be useful for most of the admins for there in oculus scheduling needs in a data center so let's look at the resource sharing models so we first start with the simple you know mesos managed cluster in this example we have a missus master running and we have three missiles slaves or the mesas agents running and since yon works closely with HDFS we assume that HDFS is present in the cluster and is accessible for yon to run me know any jobs that would interact with HDFS for example a MapReduce job and you might be aware that there is also an HDFS framework for missiles so as long as I mean as far as media is concerned it doesn't matter whether you use HDFS framework for missiles or otherwise but the essential need is that HDFS be present in the in the cluster so we start with this cluster and we launched the resource manager which is the master for young and you can actually launch the resource manager you know by by hand or you can actually launch it using marathon which is a meta framework for running any arbitrary process in this example the dotted line I'm not sure if you can see it but so the dotted line on the top you know resembles the missiles container within which the resource manager is being launched so and the good thing is that mediate can actually plug in into the resource manager process itself there is a yarn configuration for specifying what type of scheduler the resource manager should use there are like three types of schedulers like fair scheduler capacity schedule already FIFA scheduler and what myriad essentially does is it extends these the scheduler classes so that the scheduler functionality itself is still available but mediate can initialize itself by extending the scheduler so this is a very simple configuration that we have used to run myriad inside the resource manager and we will see further why we did that y media should run inside the resource manager so mirror itself has multiple components that are part of the code that is running inside the resource manager the most important part is the Mesa scheduler so remember like missiles is a two-level scheduler and the framework gets an opportunity to choose what to do with the resource offers so so we need a mess of scheduler running as part of this framework to and we have a rest interface for the admins to interact with the system and you know like there are a bunch of other classes that media itself uses to accomplish its functionality let us go back and look at the process so once the resource manager is launched myriad registers itself as a framework with the mezzos master and from this point on anything that happens in the yarn cluster have by taking resources from missiles so the next step that an admin would want to do is to interact with meteor and you know see what he can do it with her so we have a rest interface using which the admin can interact and the most imported api's our flux up flex down config and state so flexa basically helps the admin to scale up the yarn cluster so the Edmond can actually try to launch multiple node managers using this API and flex down is an API to actually bring down the yarn cluster to you know limited number of nodes or node managers config is basically an API to query the current configuration that's used by meteor and state is an API to query the current state like you know how many node managers were running how many words they were in staging and how many are you know are killed laurie norton they died abruptly so let's look at the API for launching a node manager the API looks something like this so we have defined an abstraction called profile profile is basically an abstraction for the resources that the node manager has to advertise to the resource manager once it's one set registers itself with the resource manager remember in the young cluster resource manager performs all the scheduling and the node manager is an agent that advertises the capacities that are available remotely on a node and based on the free capacity available on the node manager the resource manager will perform the scheduling so medium profile might mean something like you know like 20 gig of ram and maybe like 10 cpus so when the node manager tells resource manager it has you know these set of resources available you know resource manager can actually schedule some yarn containers for that node manager so in this example we are trying to actually launch one node manager with medium profile so when that request goes to myriad myriad basically queues it inside it's the memory and it waits for offer from missiles that can actually match the capacity that you know we want to launch the node manager with so the offer should be big enough to launch the node manager process itself and capacity for the future on containers so we wait until that offer is available and if it's not available that means there are other missiles frameworks that are of higher priority than yon that are running in the cluster and those frameworks are perhaps receiving the missus office instead of meted itself but just in case like we receive an offer that matches these specifications we just go ahead and launch a node manager and like in this picture we see that there is a custom executor that is running as part of us actually like it's we launch a custom executor that in turn launches the node manager and this is needed because node manager itself requires some configuration yarn specific configuration for example like how do you discover the resource manager and how do you configure the cgroups hierarchy for yarn itself so these are actually pre configured using executors and we launch the node manager with the right configuration and if you can see in the picture we reserve some capacity for future Leon containers and this might sound like myriad is wasting resources here because you know there are no containers yet but still we are actually holding onto resources but if you look at it we started with the statically partitioned cluster and mrs. was running separately and yarn was running separately and with something as simple as this we have started making progress into bringing these two together and with this fun like yes maybe there are some wasted resources right now but you have an ability to you know for the admin to actually you know kill the node managers in case the resources are needed for other missiles frameworks or actually if you use this wisely you can make sure that your boob jobs that were running separately in a different cluster that had high needs of meeting some sls they can use these reserved resources and make sure that you know those heard of jobs still me desolace and another advantage of this model is that when you are launching jobs you have some Headroom available for running the application master straightaway and further requests can be like you know further containers can be launched later on if you have more capacity available in the cluster so let's look at another model officer sharing that myriad offers so we have a special profile called is zero profile and what it does is it launches and node manager with zero capacity that means when the node manager registers itself with the resource manager it just tells that I do not have any capacity so the disposed manager can't straightaway allocate any containers to this node manager but why is this useful I mean you have an old manager that says I don't have any resources and resource manager can't do anything with this red way but if you look at let's look at an example like you know let's say that you have some apps submitted to jaan ab submissions goes to the go to the resource manager and the resource manager waits for hard bits from node managers to perform the scheduling so when the medium profile load manager heartbeats the resource manager it can actually schedule some containers to it because there are pre deserved capacity available on the node manager but there can't be any containers that can be allocated to the zero profile node manager but imagine like if the node manager that's you know the zero profiles node manager the slave on which it's running if that slave node has capacity available then masers can actually offer that to mediate and what media does is it uses those resource offers and it dynamically resizes the node manager capacity so if you have like you started with a zero capacity known manager but let's say myriad gives it or actually missiles gives you an offer for you know Phi Phi gig and you know two CPUs then you can actually go ahead and run a few containers using those resources so once you receive this offer mediate can actually go ahead and launch the containers for yon and yeah and the if myriad doesn't receive any offers obviously there are no containers that can be launched and as the containers finish the medium profile node manager still holds on to the resources that it previously obtained from missiles whereas the zero profile node manager actually starts giving back the resources back to my source and these resources are now available for other frameworks or back to mediate again like if there are no other frameworks that are running in the cluster so when the wen Yan is not running anything we go back to where we started basically like we have a node manager with the medium profile still holding on to some resources and the zero profile node manager gives back everything so these two models I think are useful for admins to be you know flexible to schedule the number of node managers with you know different profile levels and use the node managers with zero profile you know wisely so it helps to actually meet the SLA needs for high profile or high priority Hadoop jobs and you know any surplus capacity that is available in the cluster can be used to backfill the you know low priority hurdle jobs so let's quickly do a demo of what I have just described so I just fired up a MapReduce job let me actually show you the UI for of the cluster pulse so I have a four node cluster that has a mrs. master and three missile slaves and I used marathon to launch resource manager just like I described in the slides and this is the UI for myriad and if you look at the flex up options like it lists the different profiles available in system and that men can actually go ahead and you know like he can define custom profiles in the media configuration and they would seamlessly be shown on the UI so the zero profile will have zero CPU and memory and small profile has one cpu and 1100 gig of ram learn her megabytes of ram so we have like all of these options and you can specify the number of instances that you want to flex up and then perform the flexor and the tasks tab shows the active tasks that are running and the you know there are other options like pending tasks and staging and killable tasks so currently I have launched a known manager with a medium profile and an old manager with a zero profile and if you look at the missiles you I it actually shows three tasks that are active one is the task for the resource manager itself this was launched using marathon and these two are the tasks that were launched using mediate and one of them is the medium sized profile and the other one is a zero suspense zero size profile and this is the UI for yon and we see that there are 2 to node managers that registered with the resource manager and one of them is having non zero capacity by gig of ram and to cpu cores the second one has registered with zero amounts of cpu and memory so let's look at what happens when you submit a job so I have a small pterosaur job with I think like 10 mappers and to read users and so the first container went on to the medium profile node manager the sap master and then once the atmosphere is up we start seeing that there are other containers that are getting launched and if you look at the missiles you I we see that for every container that is launched on the zero profile node manager you are actually launching a corresponding mesos task so this is actually not physically running missiles task but it's a placeholder for running the yarn container the corresponding yarn container so when all the containers are finished we should see that there are no more mrs. tasks that are running in the cluster yes so we can look at the yarn you I to see what was the capacity of the zero profile or manager so we notice that we received some offers from missiles which we use to expand the capacity of the zero profile node manager so it's currently at three gig of memory and one cpu and i think looks like the job is done and basically like we come back to where we started that is the medium profile stay said you know what was the whatever what was the originally original capacity of the node manager and the zero profile goes back to those you know zero gig of ram and zero sequence let's go back to the slides so basically we think that this models of resource sharing actually helps both Mesa and Hadoop and the the first advantage that we see is that yawn if it can run on top of mesos it actually because of its powerful ecosystem around it it can actually help you to run a lot of these Hadoop related applications on top of missiles and as well as we have seen earlier like you know if there are tier 1 services that are running in your messes cluster then those Stearman services can actually utilize the Hadoop resources earlier in the window in the earlier model before myriad basically you had two clusters there were separate and there was no resource sharing between both of them so there are there is a hundred node missiles cluster and a hundred node yawn or Hadoop cluster and you run out of capacity in your mezclas cluster there is no way to actually borrow resources from heart open but let's say you actually have 200 node mesos cluster and you have yarn running on top of it and you have this flexibility to actually grow yarn to 100 node cluster within a mesos cluster or actually shrink it back to a 15 or missiles are young cluster and utilize the rest of the nodes for running your tier 1 services and it also helps you to do the other way around basically the unused resource capacity can be in the maces cluster can be used for Hadoop itself so as an IT admin like it's actually a pretty good deal for your infrastructure because you suddenly have a way to utilize your resources in the optimal way possible and the cool thing about this is like you do not actually require any code changes on the Mesa side it just works seamlessly on the other way like it also helps improve the Hadoop side of things firstly like you have elastic scaling earlier you started with a cluster and unless you add more machines to it and provision more node managers into it you do not have the ability to grow the yarn cluster but with myriad actually you can grow the yarn cluster to to the fullest extent possible like if you have a 200 known messes cluster you can pretty much get a 200 node young cluster and it's a fault-tolerant like myriad monitors whether the node managers running healthily or not and if the admin's says he wanted you know four node managers with medium profile then myriad monitors constantly that all the four instances are up and running and if any of them died due to hardware failure then myriad automatically starts them on another missile slave node and we talked about this like there is a pretty good resource utilization and sharing between the frameworks and the highest sell a Hadoop jobs are still not unaffected are still unaffected because you always can define the size of the node managers to be flexible enough to suit your SLA needs and as an IT admin I think it's a powerful API for for you to actually figure out like how many node managers you want and you know at what times you want for example during the day time you have clear one services and during the night time you want to be able to meet their silly needs of your Hadoop jobs so you can use the API to be you know to suit your needs and it doesn't require any code changes on the yarn side or any of the other projects that are dependent on yarn so it's a pretty good deal here are some of the other important features that the media team is working on so we talked about the ability to launch resource manager using marathon and in fact actually you can discover marreth or sorry you can discover resource manager using missiles bns so the node managers can always you know use the missiles DNS hostname and connect to the active resource manager the resource manager dies marathon being a better framework it can actually detect the failure and relaunch the resource manager on another node the node manager seamlessly connect back to that node and so one of the other things that we recently worked on was the ability to distribute the Hadoop so mrs. slave can actually remotely fetch the binaries that are needed to launch the executor and the node manager and media provides a configuration for the location of the of the binaries and as we have seen in the demo like we have a web interface and we're trying to actually expand on what you can see on the web interface we also want to try to make media production ready for folks to try to use it in their production systems so the important thing to achieve that goal is doing the high level high availability of the scheduler so you're actively working on that one and we also want to be able to launch any of the yarn related processes using meted itself so today we have the ability to launch nor managers but we are working on the ability to launch other process like job history server and the timeline server in the future using media itself and we would like to hear from you like what you would like to see in in you know us to work on and we are happy to actually listen to your feedback and incorporate that as we go along making progress on the project so there are some are some of the links for learning more about myriad we are currently on the github at github.com / missile / myriad and we have a pretty active community of developers on the wet media.com incubated order peridot and we have a hangout sessions every couple of weeks and if you are interested to learn more about where myriad is going and what features we are working on please feel free to join the hangout and we have a myriad JIRA available and the incubator proposal and the Status page are also available for you to you know keep you updated about meteor that's all we have and we are happy to take any questions sure how does the scaling work with HDFS decommission I didn't quite understand right I see so the point is so yon might be there on a few nodes and HDFS might be there on several nodes and you can be adding more HDFS nodes and how would how does it actually work together right so so ideally what you would want to do is you want to be able to run node managers on exactly the same set of nodes where the HDFS data nodes are that's because node manager can utilize the data locality better but that may not be the case always like even today like you can have the cases where node managers are not actually running on the data nodes but as long as Hadoop or HDFS is accessible from yarn you would be able to resource manager would be able to schedule containers on the node managers so in cases where HDFS is running exactly on the same notes you have better performance but in other cases you have less performance but the goal is to actually you know strike a balance between better resource utilization and better performance so yeah so I think like if you actually launch the node managers on exactly the same set of nodes where your data nodes are that's going to give you the maximum performance for your Hadoop jobs does that answer your question so I think currently what is missing in the media API is the ability to say hey I do I want to launch these set of node managers on these specific no hosts and there you know if you have an API like that I think it's going to give you an ability to launch the node managers exactly on the same set of nodes where you have the data nodes running today it's a little more arbitrary like it can launch anywhere but we have an active JIRA and we are working on that one go ahead I think the question is when does the zero profile node manager gives back the resources I think as soon as the container finish running I think it just releases the resources back so what we do is when we receive the Mesa offer we expand the node man actually we use the offer to project to resource manager that the node manager suddenly has more capacity available and that lets the resource manager to schedule young containers and but you have to also tell missiles that we are actually taking this offer and you know doing something with that offer right so what we do is we actually launched something called placeholder tasks corresponding to each yarn container and those placeholder tasks basically live as long as the corresponding container lives so once the young container finishes mean we kind of know that okay so that container finished so we relinquish the resources back to my sauce by saying that ok so the status of that placeholder task is now completed so the question is why not use zero profile always it's because you know like different admins have different needs and you know perhaps one use case is that you have some you want to reserve some capacity for your yarn cluster itself because you have high SLA jobs like you usually you have like at least twenty percent of your herd of jobs are very very important jobs and you want to be able to launch them even in case when you know there are tier 1 services that are running in your cluster so that's one reason why you might want a you know a medium profile or a high profile set of node managers but you have freed you know if you do not have that kind of need like for example you are running a few a cluster within a mrs. cluster where everything is so dynamic and there are no no needs to actually deserve capacity for your yarn cluster then you can actually use zero profile always right so the question is so media doesn't have the ability for doing the data locality yet and you know she was asking why can't media take the offer and figure out whether that offer is good to be accepted with respect to data locality the well we could do that but basically the idea is that if you have something like the zero profile on managers then they are actually very tiny they do not consume a lot of resources and you have the node manager as one of the entry points for resource manager to schedule yawn containers and so let's say you have a cluster that's running like 20 data nodes you can pretty much launch 20 node managers of zero profile there so that you can have you have the ability to utilize the data locality so that's one way to do that but I think the better way is as I was telling him is to allow the admin's actly he wants to launch the node managers that way like you can always be sure that the node managers that you want to launch always land on the exactly same knows where the data nodes are learning so it is slightly harder because the data locality decision actually happens when the resource manager performs the scheduling so when the container request comes in the container request has reference about like where exactly the containers should be allocated and myriad can try to look into that but i think that's further down the line i mean like we can we want to start with something simple and something less disruptive and you know if that does not work out then we want to improve things sure no I think so the question was like flexin doesn't seem to work sometimes and it's a bug I mean it is supposed to be working and we got to see like why it's not working so i don't know i don't know the current answer for that I mean but we are pretty actually working on it I'm sure like it should be fixed easily sure sure you can still do sorry I didn't catch the question actually so the question was what is the concrete use case for using yarn and missiles together I think the origins were like you know I think one clear use case is when you are doing you know any vector daughter company that has you know huge user-facing service like for example Twitter it has like the capacity planning for tier 1 services happens for peak traffic so do when something is trending on Twitter probably like you know you're mrs. Custer is highly utilized and during other times it is less utilized so you know it's it's when it is less utilized you cannot do anything about it because you know it is siloed from the Hadoop cluster but if you have an ability to launch a on jobs on top of missiles then you would be able to utilize that cluster better so I think that's one of the motivations I think that's the last question we are out of time I think we are out of time sorry will will talk to you thank you