Send Heterogenous EIN with airSlate SignNow

Send heterogenous EIN

[Music] thank you and good morning I'm max I'm here today to speak about the topic scaling signatu with many Hitler genius projects and still preserving configurability it's a hard to pronounce title and it's a complicated title but it's also for a complicated topic because scaling stuff is hard and so I'm here for this 11 we are managed hosting company from Berlin additionally we have an infrastructure as a service offering and managed kubernetes on top of it and we are also an Internet service provider but everything I tell you today about is about the managed hosting part of our company so we we are using signatu now a managed hosting apartment and I'm max I'm responsible for everything monitoring related inside of the managed hosting part if you want to follow me on Twitter or github my online Nick is eco I feel free to tweet about this talk if you want to before I really start I want to give you some context why I'm here today so why am I telling you all this in April I attended the Sigma camp in Berlin and it was a nice event where interesting people had nice talks but at the end of the day I was disappointed not about the event itself but I was not able to meet people with the same issues that we have with our I singer too and after that I decided ok if I'm not able to find people with our problems maybe I should talk about our problems and our solutions and our workarounds give you some real-world examples our irani singer too and then maybe some people will start to share their own stories so I'm not like only here to share our knowledge and also here to inspire you to share your experiences with singer too few words about our infrastructure we have two data centers in Berlin and we are monitoring around 3000 virtualized hosts they are all based on Linux so mostly Ubuntu or gentle and very few window servers we are checking 60000 checks and all our checks are distributed in several hundred customer projects and the projects for our customers are similar but every every setup is a little bit different so we have a lot of custom checks for our different customers which is pretty important for the job title part about configurability because we need to change stuff for different customers so in january we went live with our signals who set up before we migrated our alt infrastructure to it and the migration took us about half a year and part of the presentation will be the process of migrating our setup so I promised you a real world example that includes technical depth I guess every one of you has technical depth in your infrastructure so before we had a signatu we had a single one and before that we had Nagas and we started with one mega server in a small setup with only a few engineers and then Susie Levin started to grow so in the end we had six independent signal one service they are monitoring each other to make sure that if one server fails we at least would at least know that one of our monitoring service failed but there was no automated failover to another server the configuration was shared by a subversion repository and it was synchronized by cron jobs to make sure that our monitoring infrastructure of us up to date and our engineers if they wanted to make a change there connected to the life environment changed something they really order T Sigma and production and hope that it still worked and well I think if you want a definition of technically that that is one and yeah in an ideal world our engineers would make a subversion commit so we would have an idea who changed what and why but most of the time there would be a cron job to just commit all changes and then synchronize them to our service now what that was not idea it was okay when we had like three engineers and 20 servers but definitely not met 30 engineers then you just don't know who changed what then he singing one announced the end of life for the end of the year so one reason more to migrate to another solution and we had an interesting issue with a signal one it looked like that that the average check latency in our a single one set up and despite the fact that the phones are very small the interesting part is the Green Line it's at three minutes here it's a picture I could find from our previous set up but we had days when this was at 50 minutes or 20 minutes and it means that we had 20 minutes latency under we actually saw an issue in our infrastructure so if your monitoring is delayed 20 minutes you don't need a monitoring because you will have lawsuits all over the place and so luckily we found this very early and we were able to debug it and we found workarounds because we are good at finding workarounds so we are still about technical depth technical depth can look like that in the beginning it was a brash one-liner in our internal documentation and if you think about maintainable software that's not it it takes the 60 worst delay checks any single one and then forces is single one to recheck then it looks crazy it sounds crazy but it works it fixed our signal one issue in a way that we were able to run issing a one for a few more months and that way we had any latency in our infrastructure so we caught at the monitoring we needed to have to provide our managed hosting to our customers so that's the real world I guess everyone has similar experiences in larger infrastructures based on that we had a few requirements for new monitoring solution 100% automation I mean it's 2018 we don't want menu a chain no monitoring setup we also wanted to be high available so no six independent servers running somewhere we wanted one cluster with automatic failover we wanted to have stage setups no engineer should change the production why I meant to test if the idea works also wanted it to be maintainable so no crazy bash one-liners and we wanted a full state migration when we decided that we needed a full statement Gration we knew that this part would be hard but I think no one of us anticipated how hard it would be to do a full statement raishin of 60,000 checks from six service so for state migration means that we have a lot of runtime changes any single one like acknowledged alerts comments down times yeah disabled active checks and stuff like that it was changed in signal 1 and we decided that we needed to migrate all of that because we had thousands of modifications in the running instance and so we needed to way - yeah migrate that and later on I will show you some ways we were able to compare running a signal one to running a signatu set up to make sure that we are not losing any checks or acknowledgments but first I will talk about 100% automation so how do we configure Reisinger - that's always a question I have for other engineers like how do you deploy your configuration and signa - because there's one way manual editing then there's a second way it's a Sangha director and then there are hundred other ways only a few people seem to talk about and we have one of the other ways our complete infrastructure is configured by puppet so obviously we are using pepper to deploy and configure a signatu there are two upstream puppet modules for missing nets a puppet a signatu and puppet escena web - we are using both of them and they are pretty awesome we've wrote our own puppet wrap around them it's 11e signal - and it runs on all our monitoring server so signal master satellite signal web signal database we are using this module on those servers it will tell you a little bit more about data we brought a second part modules this 11 monitoring and we are using this module on all servers we want to have in our monitoring so we're using this puppet module on all three thousand servers and it is responsible to register every house to normal monitoring then I said we wanted to maintain configurability this means we need to away for our engineer to change settings in our monitoring environment without actually changing our monitoring environment and production so we are also using puppet for that and we are using Huayra if you are familiar with puppet is anyone familiar with puppet didn't ok so for the one who are not familiar with it IRA is like a key value store in Yama files so we are having a good repository here as fires for every customer and every host and you can change settings in those Yama files then commit them into the git repository and they will be used by puppet to deploy our configuration this is not monitoring specific we are using that for our complete puppet infrastructure ok so if you are not familiar with puppet puppet s1 advantage compared with tools like in Cebu it has a persistent state across all your puppet runs so you have the puppet database that stores information about your puppet environment and on the left side you can see all our servers we want to monitor so that's like 3,000 servers that are using the system level monitoring module to export resources into the pop database and then later we are using the sizzle Evony signatu module on how a single master to actually import those raise resources so what's the resource for example we are using the upstream signature puppet module it exports a host resource that is that represents one of our monitored hosts and puppet adds a few variables to it to announce what checks are necessary on this host so we are able to run puppet on every most registers the host and announces the necessary services I will tell you more about that part because that's an interesting one we are able to configure all the settings we need for this host so if you are deploying 60,000 checks you need some same defaults you don't want your engineers to configure every check manually in some way so we are using using other puppet modules for other software infrastructure for example we have a lot of database service based on my squared so we have a mile scale puppet module this is a snippet of puppet code and at the bottom you can see that we are using the city level monitoring module inside of our MySQL module and we are calling a function register variable like record string variable my scanner DB so this way the nice creme module announces I want to have a MySQL in your DB check and in this way all of our module a puppet modules are using sea level monitoring to register their own checks so if we have a service a host that is newly created we have about fifteen default checks and as soon as you are using using puppet to deploy most services you have like 30 or 35 checks on every host with default settings so engineers check the software and said okay we are needing those thresholds for the service we are deploying this our default setup that already works pretty great but as I said we have different customers and every customer has some needs to change stuff so we wanted to override our same defaults you're using Huayra for that so this is a Yama file and again we are using sis 11 monitoring and then we have this map of settings and here we are over writing our mysqldb check and we define another value for max check attempts retry interval or who we want to notify based on that you are able to change every setting you can change for any signal to check so this could like 50 options but it's also okay to only define one define one option you want to change that is really powerful because every one of our engineers is able to edit the siamo files and then just push them into a git repository and it will change the monitoring for this specific host or for this specific service so in the daily business if an engineer wants to change our monitoring all he or she has to do is to edit Yama file not writing puppet code or anything fancy just a gamify and then there are a lot of sanity checks to make sure that the CNF I actually makes sense and then it is deployed so we are able to overwrite our same defaults with some custom options but what if we want to add additional checks we are web hosting company primaries so we have a lot of HTTP checks and there are no real default HTTP checks of course you can check that your apache or nginx works but usually you want to know if the application behind that works so you want to have string checks and every one of that is custom for a customer so we are able to modify all options that are necessary for an HTTP check if you want to but it would be also enough to just define a hostname or IP and maybe define a string that needs to be available in the response of the server so when one of our engineers starts to build a new customer set up he or she can easily add a few HTTP checks and there's a basic monitoring for the web application you don't only have HTTP checks we also have NRP checks they are deployed in a similar way Cecille have monitoring modules used again inside of a unifier still inside of the git repository and just by that it will it will be deployed on the customer server and a normal monitoring so that is how we maintain configurability across all those different customer setups that's an yeah a summary of our export strategy strategy into pappa database of course we also need to import them on in our icing setup there's one issue it's quite complicated and we are using the upstream using a 2 module for that and when we started to use it every puppet run would take about 15 minutes and a usual puppet run in our infrastructure takes about 5 maybe 10 seconds tops so 15 minutes is not ideal especially as we decided to execute Peppard in our a single setup every 30 minutes to make sure that when a configuration is updated soon enough so 15 minutes is hard when you want to test something so we spent about a week to profile our puppet server and upstream missing a 2 module we found out how it works and we found that the module is doing some stuff to provide its full functionality but that we didn't need all of that because it's Ora set up so we can make a few assumptions about structure of our code so we wrapped some parts of the upstream module and now it includes a few custom queries into the puppet database which took us about a week to come up with it and I think it would take about two hours to show and explain it so I decided to leave it out of the talk but if we are interested in that feel free to talk to me so in the end and we are able now to run puppet in five minutes that's still like five minutes F like four minutes too much one minute would be ok we are now at five minutes raised which is not great but it's well better than 15 and we are still looking for a way to make this faster if you have any idea talk to me when our puppet imported all the host resources from our puppet database the signature configuration looks like that if you are familiar with it signature is anyone here using a signature okay that's quite a few people so you probably know what that is it's an host object in a signatu and there's nothing really special here and address a few variables for sample we have our own variables for service level agreements or off on which hardware note virtualized house is running the interesting part is this year MySQL in ODB based on the fact that our Myers came out your exported variable or a signatu import and puppet wrote this line and the configuration and that is used to apply our I say our my scanner DB check so every module that adds a line with Suzy level monitoring register variable will well it will lead to a line like this in our icing a2 configuration still this is not anything that you check it's just a variable which is doing exactly nothing so we need a way to actually apply a service again we are using puppet to build this configuration file and if you are familiar with a signatu you know this is a service especially this is my scanner DB service we are using on our example host the interesting part is at the bottom is sign where that's a powerful way in singer to to have service template and then apply it to a bunch of hosts so every host that has this variable mysqldb enable equals true will be monitored with a check my SQL any DB in ODB so that's all we need to apply this check and we have a lot of service templates like this they are in one large file generated by puppet so that's our default check with the same defaults on MySQL in ODB check we need to be able to override the settings like I showed you in Huayra that's happening inside of an template that is included in this template that includes only a few a lot of if else if else if else if else just to override every possible setting but this that helps us to ya have our same defaults in this template and then override settings if necessary and then our experience that works pretty great so now I showed you how we deploy our configuration that setup is 100% automated we can delete our completely singer to cluster and then we just run puppet and we'll be there again hopefully we tried it a few times so after we are done we are done with our migration we deleted everything and started to set it up again and it worked pretty well so yeah the requirement automation was fulfilled by that then we wanted to have a hive available set up I will give you a second to look at the confusing picture it's in two parts on the left side we have our actually singer - back-end and on the right side we have our icing upfront and they are separated software's so on the left side we are using 60 signature zones so that's a lot less than you may have in your own setups a lot of people have like 200 zones we are using only six we have one master zone which includes two masters one in each data center and so if one data center goes completely offline the second one will take over completely we tried that not shutting down the whole data center but we try to shut down one a signatu master and it worked that the other one would execute all what schedule all the necessary checks then we have two large signature zones one in each data center and we are using two satellites in each data center to make sure if one satellite fails or we have to reboot something we still have a second one to take over we would like to have three of them in each zone but it's not possibility singer - we noticed that when we tried it and everything exploded and so but usually two is enough then we have two more zones they are very small management zones with only a few hosts and we are using only one satellite in each of them course we decided if this satellite goes down we will notice that because they sing our realized that one satellite is gone but it's not necessary to have a second one but it would be easy to add one then we have another zone it's a global own across all of our signature servers and it includes for example the this configuration from the last slide and stuff like that that just needs to be available on all servers so our a singer setup usually works like that we're executing puppet on the master one then it deploys the configuration and it's synchronized by the Insignia to cluster to all other servers so the master 2 and to all satellites if necessary then the master 2 starts to dump its complete configuration into the I do dB if you are familiar with a signatu you know what a painful part of software this can be but it's well at least it works all stuff is dumped into this database and unfortunately an hour setup this takes several minutes so when we deploy our new configuration every 30 minutes it takes a few minutes until this information is available in the nice Kaido dB so in this few minutes we will have delayed information in our icinga web interface because it's in effect a signal app uses the data and the my Oscar you know DB to display stuff we know about that it's not great but we are hoping that someday in the future the I do DB role will be replaced in that and this problem is hopefully gun so I'm looking forward to the talkin afternoon maybe they will announce it but we will see so for now this works we are exporting all events of our signature cluster into an elastic stretch cluster we are using a signal pit for that I will not talk much about it because we are not using this information at all at the time we are looking into it how it could help us to analyze or a signal setup are currently we are just writing it into it and then after a few weeks we delete it so hi available on the left side I think we have two satellites and two masters in the important parts and they are scheduling checks to the bottom where we have all our customer setups so it's Cigna web we are using anycast to route our traffic to the different data centers to make sure if one data center fails we still are Vale or it's still possible to reach our web interface then we have Hardware load balancer each data center the hardware load balancers do a lot more than just ya doing load balancing for it I sing my web setup they are also in doing stuff for our customers but we use it to free sing adapt to each data center we have two we sing about servers and yeah if one load balancer fails they also other one will take over if one back-end fails the other one will take take over so that should be high available enough right height variable that's what it looks like everything that is read can fail and our monitoring still works we will still have 100% of our alerts and of our recovery notifications we will not be able to acknowledge checks in the web interface we will not be able to see in web interface but at least we can see the alerts we get SMS and emails and when this happens we will notice it because two data centers are mostly done and yeah but luckily for us that never happened and but maybe it was something so it's hi available at least in our definition of available it's enough and we are not only running one of those setups we are running to one is the production setup and the other one is a stage setup we use to test changes or to test very similar to updates and I can't recommend that enough build yourself a stage setup if you're updating your life environment it is offline for six hours that's bad if you do that in your staging environment it's okay nobody really cares you have time you can fix it and then when you know how it works or there was an upstream fix you can update your life environment so since we went live in January we had a singing outage in our production environment and yeah I'm really happy about that okay so second requirement was high availability I think that it is known you may wanna how crazy is this 11 to use n RPE in 2018 to monitor all the infrastructure there is the signatu client the signatu client has stuff like encryption and authentication so there are a lot of reasons to use that one but we are still using an epi why when we decided to migrate or sing a tune we knew it would be a big task and we didn't want to make it much bigger because we know if the project is too big it will fail so we decided hey we are having an a PE it's running it's executing our checks it works and it's only running inside of our own data centers in our own cables so the security issue is not that big that we need to change all the stuff at the same time I think some one day in the future we will replace an MP e but now it works and we could just reuse it like it was in the previous setup so just if you want not while you are so crazy ok full state migration was another requirement migrating stuff is a lot harder than building from scratch when you're building your infrastructure from scratch you can do whatever you want it just has to work if you're migrating stuff you have requirements from the old set up you have old data you have data you don't understand because some previous co worker brought it or changed stuff you don't understand anymore so it took us a lot of time and we had a lot of manual and automated work to migrate all changes like acknowledgement and down times from the a single one set up it was in this step when I joined the team that was deploying this so I had a lot of manual and automated work to migrate with parts the co-worker wrote a software to to make a diff of a single one and a signatu internally it was called a signet if and it catches all information from your signal one set ups then it fetches all information from using that who set up and a tries to find differences it can included a lot of customs this 11 specific code in the last few weeks we extracted then also c11 stuff of it and now we are open sourcing at another MIT license as the signature migration UTS so if you still have any signal one tweezing a - migration to do should check this github repository out there's one small issue if you check the website now you will realize that we had some scheduling conflict on Friday when we wanted to release it so it's not released yet I created the repository to make sure that you can subscribe to it and next week when I'm back in Berlin we will release that so a huge shout out to my coworker Ingo who wrote all of this code and it's not ready to use in production solution it's more like a framework you can utilize to build your own migration there are a lot of examples like hey if you have an HTTP check in single one you can compare to anything that a - an HTTP checking signa - like that and you may want to make sure that you check this very area and stuff like that so it's also able to migrate all your acknowledgments downtimes and comments so the signature migration your tools are able to figure out which check is the matching check in the new environment which sometimes is really hard because a signet who has a different configuration logic than any single one so there's some fuzzy matching like okay this is probably that one and if it's not able to detect which check it is then it at least tells you I don't know where this is you may want to check that manually so a lot of code examples you should check it out next week I'm very sorry for the release read delay the last part of my talk is about our modifications in singer web we have around 30 engineers using a single app and 10 to 15 of them use it for incident response management so we need to make a few changes to adapt it to our workflow that's awesome hot message if you have a tool and it doesn't do what you want you should adapt it instead of rewriting a new and if you are familiar with this signal app you may notice that this looks a little bit different we added this blue little marker it says service level agreement for this host that helps our support engineers to prioritize our alerts there's a blue marker okay I will have to call back this customer in a specific time and it was a little bit tricky to get this working but it's just a little blue dot and it's super helpful for engineers we also added additional documentation of one in another note about the SLA marker it's early I'm a saw this sis 11 SLA variable inside of the whole store inside of the service template you may remember this is used biasing a web to draw this marker additional documentation our engineers need to respond to incidents in a specific time and there's one option opened five different tabs and using all the search interfaces for our hardware node monitoring or trending or our internet service documentation or customer specific documentation that takes a lot of time because you need to use different weapon defaces in every web interface you will just search for sis 11 OMC so we decided to add links to all the necessary documentation that is super helpful for engineers because they can start to check all the docs also another change we made it's one I'm personally proud of because a co-workers who is sitting somewhere in the audience came to me and said max the history tab oficina web 2 is nice but we have a lot of stuff that there that we are not interested in it we want to know when we acknowledged an alert can we do this and I was like oh that sounds hard and in the beginning it was hard because I don't know much about anything about internals but I spoke to another co-worker who knew how the framework works and he showed me how to use it and it's about 50 lines of PHP code to get this list here and it's also super helpful because our engine is now C ok this alert was acknowledged yesterday and the day before yesterday last week and it was always at 3 o'clock so maybe there's a pattern and it already has links to our customer tickets we are using Zendesk for customer tickets so he or she can just click on it and check out of there's already an incident response ongoing with our customer or can check out what risk co-worker did to solve the issue so one last modification the acknowledgement of tickets we decided to build a feature ACTU ticket it creates a Zendesk ticket by our internal API to you know to put the complete workflow of incident response inside of a single web we are able to acknowledge your ticket we are able to group it by customer or project so maybe we don't have 200 tickets when we create 200 everyone we acknowledge 200 alerts that's helpful and then the actor ticket feature also adds all the additional documentation in an internal comment in the customer ticket so the engineer has this information also available inside of the ticket now we are able to acknowledge tickets we are able to see the last acknowledgments we are able to see the service level agreement with our customer and so our engineers are able to have it all available in side of English pretty awesome so who a few things this should be available available upstream like the service level agreement marker or the last acknowledgments yes I agree totally and the only reason it is not yet is that we need to find the time to yeah put the university parts into one place and the city level specific parts into another one and then we want to release it so I'm looking for a co-worker if you would like to submit pull requests so using about to feel free to talk to me that's always in a web set up after one year or nearly one year of running the setup the yeah the most important tip is spirit a state set up seriously if you are testing your production in signa environment you are doing it wrong in some way also automate everything you don't want any manual changes whatsoever track all related issues that's a time-consuming tip I take several hours a week to track all is similar to issues all it's in my lab issues is sing-sing Abid monitoring plugins a signal packaging that takes time but first of all it's interesting to see what issues other people find it's interesting to see how they are solved and it helps a lot to detect problems in your infrastructure so if there's an e signal to update and you check out the vector again there are like 20 issues that something is broken now you may don't want to update your own infrastructure but you can see if you are yeah if there's impact for your own infrastructure if you are debugging and sometimes I see all that sounds familiar maybe I have seen this early and the bacteria so it helps a lot too yeah keep track of all issues and it also gets you engaged in the community in some way because at some point we will start to answer to back tickets to issues and yeah so maybe we'll you will end up talking about your journey of debugging a signatu and also because it's so important please build a stage set up is it perfect well yesterday I attended a Prometheus workshop so there's some irony in there that I'm talking about is similar to why I attend Prometheus workshop and there's one thing I learned yesterday it's neither of them it's perfect so Sigma 2 is not and prometheus is not and Prometheus is not an answer for our icing a probe to a singer two problems for example currently we have some issues with signature notifications that are sent out despite the fact that they shouldn't and that's a problem that was fixed in the last release and we updated it and it's still broken in our setup it's better now but we are still seeing notifications that I'm not supposed to and I am trying very hard to reproduce it - yeah right into the upstream issues or create new ones to fix that so yeah that's not perfect but and 100% shoe if you're using another monitoring solution we would have other problems so I am convinced that is signature with the right decision for a setup and yeah I'm looking forward to make it as perfect as possible and I'm trying to give back our solutions for problems we had so yes that answers the question is it perfect since generally we had an outage in our life environment so that's pretty good I'm happy with that and my last message for you is please talk about your setups talk to each other and have you talked to me tell me about your monitoring setups tell me about your problems your solutions and feel free to ask me any questions about our setup I'm very happy to answer that but I think this is an open source conference so it should be about knowledge and we should share our different stories and it doesn't really matter if you're monitoring 10 hosts or 10 million hosts I don't really care I just want to know how you do it so if you want to reach out to me I have an email address you can find me on Twitter or github thank you for listening and I'm happy to answer your questions [Applause] have you considered allowing customers to log into the web interface or status page yes we consider that and we are still considering it but in the first step we decided not to it would be helpful because customers would be able to schedule down times for their deployments now usually they have to create an ticket like a next night we will do in deployment please down time I will set up and it would be great if they could do it by but currently not mainly because we decided to make a single app only available in our own network and we would need to manage our customer accounts currently using LDAP for identification of our co-workers and we considered an additional API - as a layer above it but it's like maybe in the future more questions sorry have you ever tried to use the API for contact changes yes I wasn't part of the team when we tried that but when we are we're evaluating the different monitoring solutions we tried it and one reason was that we wanted to be able to rebuild our complete configuration in a way we understand it so we know a lot about puppet and so we understand how the configuration files build and we are very happy to be able to debug that also when we started to migrate there was an issue in the signatu api like when we are doing a lot of changes or asking for a lot some time it would just stop and we would have in invalid response I think you fixed it recently but it's yeah it should work now but that was also a reason for us like ok maybe we shouldn't use the API for our core infrastructure put notifications and things like that how do you manage on call times and shifts and who gets what notifications when we have an internal API with tools to define a first level and a second level and the third level and singing as always just notifying the first level contact and then the notification script will figure out by our own API which one to notify so that's not part of using a tool but we have about 400 customer contacts inside of a singer too and using that who we're sending SMS and emails to them if necessary so we have a lot of notification objects in the configuration I saw in the web interface that systems you have virtualized systems that there there is a variable that points to the hardware they're running on are using VMware and are using the possibility to shift them across the hardware no ok we are using virtual it's a container environment and yeah we are asking again our internet API to ring the puppet run for all met pattern for all your matches of one host and which hardware is running on so if you are migrating one host to another it's possible that our information in a signal weapons outdated but it will be refreshed in 30 minutes and I also saw that there was a line about trending a link without running what do you use for trending we are using sabich's alright thank you there's another one I'm just curious you're using I do as middleware between your monitoring subsystem and the web interface to that monitoring subsystem are you using the I do information for anything else or is it just sort of a temporary warehouse just temporary voicing on have you considered using live status to connect the two ends and just losing the I do database entirely no but oh that's an interesting idea it was a complete shift for us so what does it called live status okay mattias Kettner and his check MK project life status basically I think Nagios has actually brought it back in stream as well but it was originally developed as basically an sql-like syntax that allows you to do to talk directly to what's going on on the daemon and it is supported and icinga - okay yeah that sounds interesting thank you you said that you have around 30 developers are they all developing no they are no developers they are admins in so they are work sorry people who are developing checks no no it's mostly me responsible for our monitoring and we have an automation team that is working with puppet and so most the people of this team working on checks so we are like I just wonder if they're all working on the same stage environment well our automation team is working on the same yes and sometimes we are using you regen boxes to test stuff but if you want to actually test our own environment we are all using using the same one and how that is usually not a problem because we are not all working on monitoring tricks at the same time so we just asked like does anyone need this environment now and then we test stuff okay you reported that you have this ideal performance problem yeah we had the same problem while we are reading the I do database from the thing about to fronted because we have over 1000 different groups because every project has an armed group with an Orion notification objects and we had heavily heavy performance problems with the read performance so we know caching okay and do you experienced the same well it's I'm not really sure but we have also a lot of groups so that could be related but I think it's more about writing into the database than reading from it okay because we had reading performance issues because we have lots of groups which are only allowed to see specific checks check results I so you mean groups in a single yet sorry filtering text no we don't have any groups in this thing about okay that's the first time okay well then thank you for your time [Applause] [Music]