Add Watcher Placeholder with airSlate SignNow

Save initial watcher

organizations are storing massive amounts of production Leda and elasticsearch the production data especially with data streaming in real-time users often want to get notified when something changes in their data according to predefined conditions the application performance monitoring as an example we are collecting performance metrics such as response times and lead and sees that a surrogate measures for user experience in this case you want to get notified when these performance metrics drop below quality thresholds another common use case is security analytics where you want to notify a security analyst when a security incident is detected in your logs one such example is seeing multiple failed login attempts from the same ip in a short span of time these are just a couple of examples with real-time alerting matters watcher which is the alerting and notification feature for the elastic stack enable feeds use cases and many more in this introductory video series which we are calling watch a lab will show you watch our inaction using a variety of simple examples we'll start simple in this first video and touch on just three things installing watch or creating our first watch and I will finally wrap up with the watch or execute API which is a nifty testing and debugging tool so with that let's get started we'll install watcher into our existing elasticsearch cluster this involves two steps first we need to install the license plugin and second we need to install the watcher plugin we do both by running the bim plug-in install command in our elasticsearch install directory in this video series we'll be working with elasticsearch version 2.3 now I have successfully installed watch let's restart elasticsearch and hop over the cabana in this first video we'll be working with Apache access logs which are streaming in elasticsearch cluster in real-time here we have them visualize in cabana each log has information like the requested URL at what time the request was received the source IP the location of the requester and much more a goal here is to detect and log attempts to access the Etsy password page on our servers such web attacks can be detected by inspecting the requested URL in the web access logs running a simple text query in Cabana shows me that there were quite a few attempts in the last one hour so let's get started with creating this alert watcher provides a restful api to create and manage your alerts in this video we'll be using sense to interact with the watcher API since the Cabana plugin that provides a console for interacting with elasticsearch api's to set up a new alert we need to create a watch a watch is essentially a single alerting rule and live composed of five parts shown here in the blank watch template the five components of a watch are trigger input condition transform and actions the first component of a watch is the trigger this defines the watch execution schedule what time or how often do I want to run this watch here I'll choose to run this watch every one hour using the interval option watcher provides many other options to define your execution schedule and we'll dive into some of those options in later videos next we need to define the watch input the beauty of watcher is that if you can search for it you can alert on it I want to my logs generated in the last one hour that had the text Etsy password in their request field when this watched executed any documents matching this query will get loaded into the watch payload I'm running this query against the Apache access locks index I'm using a compound boolean query here to look for any documents in the last one hour which had the text Etsy password in the request field if you have use elastic search query API in the past you recognize that everything within this query block it's just a regular elastic search query the next part of the watch definition is the condition the watch execution engine were test the watch payload against the condition to decide whether an action is needed or not in this case our condition is rather simple we want to alert if there's even a single attempt to access the Etsy password file in other words we want to alert if the number of documents or hints in our watch payload is greater than zero I'll use the compare condition block to achieve this the next part of the watch definition is transformed this is an optional piece that allows us to transform the payload prior to taking an action we'll learn more about this in future videos I don't need the transform block for this particular watch so I'll simply delete it finally moving on to action we need to specify what needs to be done if the watch condition is met you can do many things you are trigger an email push notifications to slack or hit chart or pager duty or simply index an event back to elastic search in this first video I'll keep things light and simply log a text message to my elastic search logs that's it a watch definition is complete we are ready to register our first watch using the foot watch API will call this watch password file alert now because I've specified my trigger to run this watch every one hour the first watch will not execute until an hour from now but I don't want to wait an hour to see if this watch works is expected and the execute API can help with that testing the execute API allows you to execute a watch outside of its normal trigger schedule it's a great way to test and debug your watches now that I've executed my watch let's go to our terminal running elasticsearch and see if our watch generated a log you can see here that our watch worked as expected and an alert message was loved to elasticsearch a current watch message only mentioned that an access attempt was made but does not indicate how many such attempts were made in the past hour let's go back the sense and tweak the watch action to include that number I'm using double curly braces here to pass the total hits value for my watch payload to my message text let's run this again to update the watch the put action is acknowledged and you can see here that the version ID was updated now let us test the update by re executing the watch using the execute API and if we go back to the terminal we see our modified log message which includes the number of attempts that's it in this first edition of watch elapsed next time we'll dive a little deeper and see how to bring in analytics and aggregations to create more complex alerting rules stadium in the coming weeks for our next video thank you