Add Heterogenous Default with airSlate SignNow

Add heterogenous default

hello everyone thanks a lot for attending our talk I am the club Debnath and this is my colleague wilderness we joined he will present this talk titled fast log analysis made easily by automatically parsing high resonance logs so let me give a true set the context first let me give a quick primer on the log analysis log is everywhere any system you are an IOT software system computer systems smart cities is a nuclear nuclear power plant auto production line every system underlying like collect logs and these logs usually have a system status so what happens is if you analyze this log you can see what is going on and you you you will be able to travel to the systems so one problem is I learn the velocity the number of blocks is very high so it is very hard for human to analyze these logs so that is why you see there is a lot of interest in the log and Alexis tools and that is why spunk is making a lot of money a law against a log its test a lot of people interested so this is one example of log analysis what you can do so suppose you have a raw logs and now log analysis can find some interesting part on that and can do multiple things so I put four different example when one is like new log pattern so suppose you always see like system is normal if something bad happened probably some new pattern can show up so this is where we're saying like new look better and another thing is the tail log event so suppose you are always getting like IP address requests from two different machine suddenly third machine is showing something so that is really it probably you want to cast it and log rate change it is like if there is volume change in whatever you system is generating so that may also indicate some problems another area is like log relation so suppose someone always log in and spend 20 minutes in this mission machine and log out but you see there is some changes happening so I put for example but lot more possibilities are possible so that's good but if you want to do any kind of log analysis one of the core step is like log first and this is a very essential part why so I put one example log here it is like saying like sometime in March 3rd some machine has some other information so as you see it is unstructured and there is no fixed format so what a parsing does it tried to make sense out of that so one example could be parsing can do is saying that this is the message itself and this has happened at this time and there is some field called PDU it is the name and there is another field like leg it has some value and some amps so it is very easy now for human to consume and make do something useful but as you know like this is not like happen magically so you need something that is we call parsing pattern so in this case like pattern could be something like that day time followed by some text and then the PDU there is a notice space lake number M's number so this is some varying field so whoever work with log they know this is not a very fun part to write this kind of like regular expression or this light this kind of pattern and why it is very hard sorry for this small text here so I put four different locks for different system so if you look on the on the log system that is not in a form form it every system have their own way to like generate log so the first logs are coming from a syslog say good one from MongoDB third one from OpenStack and last one in some custom application so if you are lucky like suppose his log of MongoDB or OpenStack someone already writes something for you so you may get started but if your in the last case which is common for like lot of people good luck you have to spend like significant amount of time to generate this kind of patterns so in our view there is three kind of problem when you try to write a pattern generator first is variety second is velocity and third is like domain knowledge like limited limited like domain novice lot of time like people who are dealing with log they may not fully know what is happening so they have to do some trial and error so what is our goal here we want to make you like easier best we want to generate patterns without or no human involvement so we may need some human help but we want to minimize as much as possible so this is the outlet of the rest of the talk first I will describe some algorithm and technical detail and then we will take over and he will show you next share with you his experience from this tool in real life deployments so fast part so if you like problem definition this is once life of the problem like definition input will be set up logs an output will be set of patterns which can parse these logs and we have two optional input 1 is tokenization delimiter and 1 is Mad Max pattern limit so there is a nice saying like you may need some human involvement that that is optional there is not necessary as I go through here it will be more clear like why these two may be in some case you want to input to the system so before even going to the solution let me put a simple act like example log so this log is showing like certain time some user is log into some machine then he is connecting to some database he is disconnecting and he is logging out and this is repeated for 3 different user he was a beer 1 to be at 3 4 and beer hash one and now we want to pass this kind of block log this is a very simple example so if I ask you what do you think how many patterns can be possible from this scenario so what answer I usually get some people can say for because logging in logging out database kind of database this correct some people say do because logging in log it out in the same machine and from the database Anakin disconnect is another even some people say it's only 1 because everything is kind of same and this part is varying so there is no one is wrong it depends on like his expectation of what you want to do everybody is right so what we want is it possible to give a option to user so he can cherry-pick instead of generating only one set of pattern we want to show him all possible option then he can pick whatever it work best for him so that is what I am saying so let's say this is the input we have and we have some kind of pattern key we are saying this is a T it will be clearly data so it's saying like if you have this kind of log one set of pattern can be like this babe are booked then one cell can be for different pattern like this and that's it another set can be these two ever another can be only one so now you saw all the possibilities and I think whoever work on the log they will agree that it mostly cover everything now if I show this now you life is simple peak whatever you think is the best for you and it is much better than working with a lower log it's kind of a synthesized format so this is what we're calling like pattern key so to do this kind of thing we did some research and we work writing last three years last year we published a paper called log mind it's like fast pattern recognition for log analytics what it does is like it show you even when you give a set of input logs how this kind of key can be generated and this is this has a lot of good properties it as you will shell we will show and we'll explain just super fast it consume very little memory and and it's very scalable so one of the things like this algorithm do with really good thing is like because it knows that it is working with the log data so what happens is if you go to the machine learning lot of time they make a lot of we get assumption if data is very random I think it is very hard to do anything special but if you think this log will be generated by some computer program which is written by some development then randomness is not that many so if same line is generating same kind or blog usually they should be very very similar so if you come with this hypothesis as we will show things become much cleaner and simple and it is at the end of the day you will see it looks a very silly problem to solve so this is the log line workflow as you see input is like you set up an ax logs and this is optional you can put like tokenization delimiter and next pattern limit but you don't need to then internally it fast do a pre-processing step then it do some clustering and from the cluster it generates some pattern this is one set of pattern then it is adding it to the pattern T it is increasing the clustering like parameter it go back and repeat this process go on until there is only one pattern lift so whenever we say clustering anything people get very nervous because some mastering algorithm requires some parameter if you use k-means your pin okay in this case how many patterns and a lot of time people don't know and some algorithm require you if salon or some min points and people don't know and we found it is very hard so the good part is in this algorithm you don't need to input everything Elega rhythm will is internally start with some simulated distance and it will type it will increase it and iteratively so you in user case he doesn't need to do any or every the will figure it out so that is very good part so this is summary this three step fast sorry fast system with the P processing then there is the iterative process they generated the pattern tree and finally there is output output step where we choose the pattern set so to explain the P processing part the example I show you here is a depth at 12 logs so in our in inner algorithm every what is a token so as you see this date will be a token time will be token login is the one token I paid this is a one token user BR one p is a one token so this token is very hard to computer to know what it is so suppose in this case we want to detect also in our pattern user separately in this case we need some little bit help so that it says that is the function of the tokenization delimiter in this case if we say tokenization nickel delivery is equal what it will do it will basically split this token user equal beer one to two into three different parts three different tokens B are equal beer one two and it will do same thing for other logs so fast step done now second step SiC what second step is doing it is doing some data type identification so as I'm saying like if you even like this log is like very honest exit but it is written with some computer program and we're there whenever want to put something meaningful that is why he can later debug so if you go with these hypothesis but what you will see from the past log this this is always changing date time then there is some login this is a English word so maybe user developer want to do in first or give some information so we'll keep it intact and then the IP address so IP address is changing so we will mask it saying okay this part is rightly address then it will keep user because it is a dictionary word and equal it is symbol but B are 1 2 this is again a mixed it is not a text it is not a alpha consists of alphabet so we will mask it - sorry - what so that is what we do for the fast log we'll do similar things for the other logs because if you do the second step now it becomes much much cleaner so whatever it like reality is in the in the input locks but after doing this it becomes now you can think like normalized or like the randomness is kind of like accessibility now we go to the the rest part so we so so far I show you how to do the P processing or the clustering part so this is the part we will generate the pattern G so we have 12 blocks we need the pre-processing this is the fast yeah clustering in these locks you will get the label one of the three so in this case we found that among the 12 only this 8 different thing exists now so first level of clustering is done then you take this thing as the input for the second level and you generate like second level of the pattern is output so as this figure is showing in the second level what he is doing it is taking like pattern 1 and pattern 5 and it is marching if you look pattern 1 and pattern 5 everything is same date time login IP address user equal only this is what a notice space so it is it is merging it and making it not a space so what it is doing as this goes up it is making things like more generalized now next level again it will do clustering on this now in this case it is merging nine and twelve so is you see nine and twelve everything is same is except I log in and log out so out of log in and log out now it will March it will generate one pattern and finally so you have now thirteen and fourteen one is from log in log out much together database connect and disconnect marks together so you get level three in levels if you see everything is kind of fixed only it has extra database so that means you can put a wild card here that a you have one final pattern so this is the final like pattern tree as this example show it's quite simple once you understand the details so so in our algorithm like that memory memory uses depends only the size of the level one because as you go down that it's decreasing so that's not a problem at all and and this is only the the costliest part to compute and other part is pretty simple because even like you do some complex computation but number of log is very small so if you think intuitively what is happening suppose you have millions of log but in level one you just boil down to eight different things now whatever complex thing you will do it will not like blow up your CPU use or like running time and we will show example like this is usually the case so in level one looks like even like you don't need clustering I think if you're a smart and I'm you already figure it out because you can do some more clever things than which can which will be able to have eight clustering part so if you think about that after doing the pre-processing part things become like that now it's like a very simple problem it's like you want if you identify the unique lines that's it that's the level one and if you still don't try to think what it is so think about every log is a different color so in this case if I think it belongs a different color now my problem is I want to identify number of unique colors and this can be done pretty fast if you have a hashmap or something in this case even you can make whole log line as HP and this can be done super fast so previously Isola Bella one was the most complicated part ice but I show you how can you do it like in a blazing fast way for the class turning but I put a hypothetical example so let's say our plastering but we have to do some similarity so suppose our threshold is point zero one and initially this is a log one there is nothing there so we'll make a cluster out of that when the log to come we'll compute the distance in this case in this case like distance is point zero two it is more than my threshold I form a new cluster now if you want to see how that class this score is calculated so I put an example suppose log one is like this log P is this in this case similarity is point zero eight what it is doing is starting from the left it is trying to find the common things and the maximum length so I put a formula under this if you like formula you can see what it is doing on and finally in this case distance is point zero two so we will calculate distance for all logs in this case suppose log three we calculate the distance and we found like with logo and distance is point eight so you can it cannot go that that cluster but second one the distance we found it is zero zero one so you put in the second cluster and same thing we do for the log for instance it is like zero one with a log one we put put there when log point five comes we compare it like the log log one it is point zero two so it is not like less than threshold we continue not 0.5 not this one continue and we put a new cluster so this is one optimization here because when we compute the clustering we don't need to compare with everyone we just compare with that the head of the of the cluster so the intuition is the following the clustering algorithm is good inside the cluster everybody has the same like importance so if you compare only the top element that's good enough you repeat this process so suppose finally this is the cluster has formed so now what we will do out of one cluster we will make one pattern and for that like the algorithm we use called like smith-waterman algorithm it's simply like I didn't edit distance bits dynamic algorithm so what it is trying to do fast you give two different patterns you try to align them so suppose in this example you have log one date time what IP address user equal notice space and here's a log to sew up a alignment you found that everything is same except this part now you just when you merge these two and you put a while cut there and and in some case maybe you have order naughty space then you will replace by the more generic one so this is like the declawed the pattern definition part so if I recap we give 12 different logs and finally we have a key like that level one has this 12 8 patterns level 2 has 4 level 3 has two level 1 has 1 so now this is what the next pattern limit comes so sometimes suppose the user want only 4 better if you say that what we will do will pick a level who what can satisfy his limit so in this case like level 1 has 8 level 4 2 has 4 so we just put level 4 as the final output because that is what user 1 and maximum is 4 and and we don't output this level 2 or 1 because they may have wild kratts so since we can satisfy you the limit using the more specific one will not we will not go to the upper layer so suppose the user deal doesn't give any input so that is normally the case because initially user may not know how many patterns you want suppose you want to go there some default so in this case we have to do something so what do we do we calculate some cost per level but this is how it is done as you see the cost for this level is 0 this is 0 this is 0 this is 12 so you may thinking how it is it's calculated it based on the number of wild cuts so in our algorithm every level has same coverage so even level 1 whatever 8 pattern I have they can cover all they can pass all 12 blocks even level 2 they will be parts all 12 lakhs level 3 all 12 blocks even level 4 so in this case as you see this level doesn't have any wild cards but the initial cost is zero but level four has one wall cast and it can cover 12 lakhs so that is a Costas tool and I think you study like Jamie has shown like this while fair is greedy data is usually very costly to compute so we try to somehow like like capture that in in this algorithm and and so in this case if either give no input we'll just pick the level two because this is like where we have minimum cost with no wildcards this is a default output but user have the option he can still cherry-pick so this is like the foot two different case if you give a pattern next limit what will output and if you have no preference what we'll do but will always output this pattern tree so you can even say if acute parents it so given this pattern said now you can you can take it and import it for the log estate and other tool and now we will take over he will show you what we did for the log is test case scenario okay I'm gonna go over some demo use cases I am on the operation side of the labs and our mission is of course the support research in any way we can and off times that involves dogfooding and so when i heard that bit blobs research department was doing something with log analysis which you know of course we do and that they were using logs - which we had started to use I became interested approached him like hey can we try it out on some of our data and we came up with a couple of use cases I'll go over now first of all let's look at the pipeline and discuss that I take a representative set of input logs and I delivered it to him I can also of course optionally say like here's how this sub tokenize these logs I have some key value stuff and you know so we'll see that and if knowing it's what the data is if I think I'm going to get way too many patterns I could say hey cap it off at this number I want any more than this number of you know croc expressions that goes in the log mine it generates its set of patterns which is an intermediate format because we're target targeting logstash that is a like a post processor that's going to write my log stash config it can be informed to the dotted line here at the top it can be informed by the input logs to set types on the output format like especially for numbers you know do in sand floats so I can do some metric stuff and of course that results in my logs - config file so the first one is a simple one I have an environmental control system that does a variety of things with having to do with environmental controls around our data centers it among other things gets all the PDUs report every five minutes that amps per lag and what I want to do is I want to chart that as metrics and see you know are things increasing you know how many amps per legs if we have new equipment to rack which rack should we put it in which leg on the power strip should we put it in so I gave a selection of almost 15,000 lines I said it has some key value pairs as you can see so the sub token ation delimiter is an equal side it resulted in one pattern took nine seconds to come up with that and it's hard to read but there's the resulting log stash filter expression has a little pre-processing and then it writes my croc and then it removes some extraneous data I plugged that into my log stash and of course you know I can go to the system and specify you know hey I want to look at the APC data in it so apologize this is running a little slowly on this VM here but anyways I can get the data it's all tokenized and I can go and draw some you know the beautiful graphs and I didn't have to write any croc you know this system did it for me in you know nine seconds 15 seconds I just take the output plug it in kick logstash and I can get to doing what the thing is I want to do without having to mess with the writing expressions you know which we all love second thing use case I gave my was thinking what's the ugliest log I have and I run this business process management software that has like a truly insane logging and one of the insane parts is you know it logs all kinds of things but this is some developers idea of an awesome log timestamp we're going to assume the month and we're going to assume the year I'll just put in the day of the month and the time and I need to like correlate that to a sane timestamp I don't know who came up with this this is three days worth of data I get almost 250,000 log lines I have to say ok it's got some key value so I'll equal side but also I need the brackets as a delimiter to like pick out this timestamp because I didn't put a upper parameter on it I did not inform it like hey I only want so many patterns it returned that set the highest set without the the wild-card which happened to be 339 in this case it only took 27 seconds to write the resulting filter and here's the whole set as we can see you know it took 339 is the the lowest number with the zero cost but I could have picked any of these you know if I wanted to cap it off at 50 I have that whole set that I could have taken the resulting filter lines and plug them in you know just just saying I didn't tell it so it's gonna give me 339 that's the default and so it was able to write a parser for it and even make sense out of that insane date stamp and as a result I do get usable output because it's figuring it out as the machine figuring out of course it doesn't have like it doesn't know what the date is so it generates the labels itself the pattern every pattern gets a number so it's pattern 199 and then it just you know the numerical fields ahrefs not spaces or NS and it has the number I can go back of course input like human readable labels just by editing the Kampf but in this case I just need due to some data exploration and see what I have but I can trace things through the GUI DS and I can kind of make sense of the flow of the application whereas before it's just like this mountain of data you know so in summary log mine is a fast concealable system that really can help take the drudgery out of doing log parsing it works with no or at least minimal human involvement it's just simple parameters if a human's involved but you could just give it with the raw data and say run very quick it's flexible because it returns all the sets it computes and I as a user can select the set I want to use on the output and NEC corporation actually used it in a shipping product internally for its customers to do actual log analysis and it's tested in the field that you know we dog food it's like a little but they've used it on millions and millions of log lines it's very fast it's getting the job done there's the link to the paper the slides will be online I'm sure and thank you for your attention and we'll take any questions [Applause] hi sorry so is it is it open source so can we can we use it oh it's not Willy's build but in the paper it has more detail within the presentation we I already I think gave little but it is not open source or there is no like github code okay I I kind of pushed him to see if it could be and of course if you're dealing with a huge corporation like NEC it would take a long time to get it through the pipeline the core the core IP in it they may not want to release the logstash bit probably but you know okay thanks