Pipeline scada alarm management for Legal Services

How to create outlook signature

[Applause] it's good to be back again at s for that I think this is my 8th year in a row speaking so thanks to Dale as well a little overview about this talk it's we have all these gadgets and gadgets that a lot of the conference has over the years been talking about network security monitoring all these sensors what we're going to talk about is how to detect these threats but the problem is you have these alerts and alarms and we don't know what to do with them there's no blog posts about this there's no papers about this so how do you go about doing this so we're gonna talk about the overview here for the talk remember threats and risks aren't going away we're gonna have a section on detection talk about philosophy of tuning Security Alert engineering is similar to ICS alarm engineering is anyone ever been an engineer for control system and had to do tuning anyone okay we're going to dig into the ISA 18.2 and e-mu a-19 one alarm management standards and we're gonna apply those to network security monitoring sensors for control systems and then we're gonna get into a little bit about the nist SP a hundred - ninety-four standards on tuning IT systems and we're gonna apply some of those technologies and techniques to tuning ICS as well and then lastly what's good about detection without response you have to have a response so I'll briefly talk about Incident Response playbooks and following your plan ultimately knowing your systems is the most important tool to operate and defend your system how does my system work what are my threats and risk of all types not just cyber do not have enough visibility and do I practice my plans so keep this in the back of your mind knowing your system is the most powerful so a little recap some of my colleagues had spoke last year Sarah flukes from Germany she had layer blueprints for ot security if you haven't watched her talk on YouTube I have a link there and her paper security needs to be engineered just like the control systems are engineered we can't just let you just put in anything without thought so she presented this really great watch light tower design model for engineering control systems and then Nathan Wallace spoke here on stage to about making power systems cybersecurity part of the engineering process so those are some of the building things that we have the talk yesterday from jake Brodsky about securing PLC coding building engineering into security not building security into engineering we're gonna do it the other way around so here's the problem with ICS security alert management there's a little published about it asset owners have to learn by doing things just like Jake said yesterday we're not taught this it's not published there's no books there's no blogs there's no nothing so here's the theory ICS alarm management is well-defined idea alert management is well-defined ICS security alert management must be engineered by combining both of these theories and practices so let's create a reference that standard izes and combines these key concepts from both to empower ICS security teams and asset owners so a couple of about a month ago Rob Lee and little Bobby posted this and he was trying to steal my thunder but he I don't think he knew I was giving this talk so let's get some Network visibility a more secure reliable and safe environment what are your goals great and what you what Mike what you think about the in-state deliverables with that scenario that impact those goals think about the end state and work backwards make sure your response scenarios are just support about detection and collection strategies so he's getting to the right point you have to have an end goal which is the engineering part an engineer the detection and response strategies and that we're going to talk about that so is anyone familiar with ISC 18.2 anyone we've got a few people okay the primary function within alarm systems is to notify operators of abnormal process conditions or equipment malfunctions and support the response so if we look at this alarm system data flow it should make sense now if you're an engineer for control systems if you're a security IT professional I added the overlaid the security logs piece and sending those security logs to the sim or the arishok if your saw countless so you can see where we can take high say 18.2 and already kind of poured it over into and you reuse it and repurpose it for tuning alerts for the sock and this is very this is kind of the gist of everything we don't have to reinvent the wheel nist SP 894 if you look at the you know intrusion detection I won't read the whole thing it's very similar into the definition of the function within alarm systems management so if we look we have the sensors in level zero sending sensors data up to the operator HMI alarms external you have your historian you just overlay that grab whatever logs may be available in your sis maybe the user logs or the event logs or the error sis PLC's Artie's already have these built in and send them in to your security log box that's separate just like your alarm concentrator so it's basically in that same key concept and in HMI it already has windows event logs Windows security logs and probably antivirus logs and you could send those up and correlate all that together where should you collect and where she did you detect threats and risks to find the goals that little Bobby was talking about and drive your security alert philosophy so operations the key goal there is monitor the process and assets safety regulatory whatever ever kpi's you may have security wants to monitor network and SS for malicious activity safety ultimately you can't see we aren't looking so if you're looking with the alarm process that's what we've been doing since we've been engineering control systems since day one when we do for engineering forensics and root cause analysis we have to rely on those alarm logs from the system and that's how we piece together the forensics what happened like after the Superbowl blackout that I got a phone call was it cyber let's start looking in the log so I had to throw up in my laptop Patrick Miller is laughing because he said he tweeted get to work Sistrunk you have to dig into the law the logs and see what happened and you might even have to pull the SCADA historian and and and see exactly what happened after the 2003 blackout there wasn't enough one of the key recommendations from that report there wasn't enough forensics data in the log so they had these wig wag charts there were pins they would write you know what megawatts mega VARs and voltages was going but there doesn't get down to the millisecond level it's like per day or half a day or maybe in an hour so there was no way to figure out the root cause so the the forensics piece you had to do the more detailed fault records and more did dist you digital file recorders that looked at the system and put sensors in the network now same with a security side you can't do forensics if you don't have any of that information so you have to do digital forensics they're brothers and sisters root cause analysis engineers to know that so when we talk about security and control systems it might make sense to say engineering route cost analysis includes digital forensics - so when they called me and said is it cyber we had to rule that out was it cyber at that time no it was not cyber it was a really miss operation that was found by all of the relay engineers that did the root cause analysis but we had to consider was it cyber so this alert philosophy you have to define your basic definitions and tie those to your operational definitions based on the objectives and personal principles for alarm systems so let's move that down into create a document for your ICS security alert philosophy define your security operations for your control system define specific alert categories and priorities define and measure metrics because the you know the the common saying is you can't manage what you don't measure if you don't measure it you can't manage meant and manage it so align these existing philosophies between your IT alerts and your ICS alarms and also your IT security alarms as well and your eye sees IT security alerts so e mu a is a similar standard to is a 18.2 mainly used in the UK and mainly used in oil and gas and chemical so it's engineering equipment and materials users Association this checklist is free so I'll put the link to it below the standard and the is a standard are not free they're not terribly expensive so you should be able to get those with your work budget if you need it if you don't have it already but now what I did was just simply let's change alarm system clearly define the intention of the alarm system let's do the alert system for the security network security monitoring sensors and basically you just over lay anything else like this define any technical terms use IOC indicators to compromise engineers don't know that today most likely in the real world but we can define that so they'll know what they're talking about Network attacks and worm what does all that stuff mean an engineer may not understand those terms when the engineer understands security as being in bounds not as cybersecurity we still have that issue in the real world today lay out the methodology for managing your alert systems management of change anytime a system has changed especially on the network the setting of incident priorities guidance on the selection of rural settings so all the snort rules and sericata rules or any of the manufactures that's and the vendors that's out there on the on the sponsor stage all of their products that do all this stuff - and they probably have a process for you know tuning those alarms tuning those alerts and make sure that you tune those things and we'll talk about that in a minute and then all all the way down to training and auditing it's really important to have that in your policy for your ICS philosophy so here's another graphic from is a 18.2 it has the audit and philosophy loop so if you if you don't have an ICS security tuning alert philosophy you really don't have any rhyme or reason to finish the rest of the cycle because this has already been done by engineers when they built the alarm system when they had to tune it so you identify what you have your rationalization like what Sara flukes talks about you have to have the engineering rationalization of why you built that the engineering part of the system detailed design implementation so that's really that lighthouse piece that was in Sara's talk and your management of change is important anytime you change it you have to have a reason now monitoring and maintenance we overlay the security ops in tuning and hunting over the other loops that was in the system so this 18.2 an audit it's it's still important but this whole process it's a circular cycle and every time you have a new change any time you have a new requirement a new threat a new sensor we added a new piece to the network you know we found some new i/o sees from like say ics-cert seiza send out there's a campaign targeting your system then when you have to inject that into your design your implementation so you're in your security ops all the way down to hunting and auditing so keep that in mind as you build out your philosophy now a recap from s4 15 talk NSM collection for those who may not know basically you're taking different collectors and putting them in your control system network so you have Network sensors which is you know your switches and you might be able to do a spam port hopefully you're familiar with that or getting some firewall alerts also you're in your DMZ your a DMZ or ICS DMZ you have enterprise collectors that may be there like antivirus or an agent security agent those are starting to be more common now it's more problematic to put them down in the plant level so HMIS you may only have existing logs some control systems are starting to put agents in there like antivirus and other instant response type agents and then down in the control level you'll have logs only so the user log event log the air log all these things that you can collect some PLC's have syslog on the NSM detection piece you know it takes these detected anomalies and escalates it in their IR process so what alerts are there what anomalies are there firmware updates you know login with default credentials or devices going online or offline or you got TeamViewer trying to beacon out and is actually account you know an active connection those things that you have to think about in your collection and detection so tuning create and refine reliable IDs rules now when you first turn on your system you're gonna have to tune it just like you tune an engine it may not run right right out of the box you're gonna have to work on it so actively managing your ICS Network sensors is is really important has anyone ever seen this alarm management graph going from the you know plant state normal to disturb too upset to shutdown and nuisance alarms standing alarms critical alarms operator priority is over here and you have nuisance alarms not so bad not gonna hurt the plant these critical arms could shut down the plant that's the kind of rationale here you have injuries and deaths emergency shutdown cut in environmental violations like a release of chemicals now part of that standard is from blog posts here at automation calm white paper on alarm monitoring management keeping the peace and quiet so really great article I just hope you go read it this also this graph was how many number of alarms per hour can the operator do so same thing a security analyst can only handle so many alerts that come in is a critical alert lost in the mountain of nuisance alerts think about it same with on a control system if you have too many alarms gate alarms the operators are not going to trust it there may even go into manual mode they won't trust the system so you have to apply the same thing that we did with the engineering of the control system alarm management we can port that over directly to security management so reducing nuisance alerts is important how many of you have ever installed a network monitoring system and like like the like bro I know Blake has a turn on default thing everything broke it alerts on everything like especially a team p3 for instance um when you turn it on it doesn't care if it's a good or bad it alerts on every single team p3 message that's just too much you don't want to alert on everything so it becomes noise so here's a graph from a chemical processing dot-com article by specific doing a an analysis of before they reduce their nuisance alarms and then after I don't have a graph for this for any ICS NSM tuning because I haven't worked at a asset owner a long enough time to be able to get this data if I still worked at the power company I would already have the data but you know I'm going from place to place as a consultant so hopefully you will be able to do this if you have a ICS NSM and when they reduce the alarms you could see the big spike in actual data so in the top a process alarms the operators actions now at the after they reduce the alarms the operators actions match the alarms there's a cause and effect now here's another one from the National Fire Protection guides for reducing nuisance alarms and you know how many's been in a hotel and the fire alarm goes off and then I nothing happened some kid just pulled the alarm or it was a candle you know went out just one little candle and a little tiny bit of smoke set off the whole thing so really have to avoid the false alarms so we can more trust the system this is a study from the UK where they had residential smoke alarm presentation on reducing alerts so we can take what we learned from that as well now I have a gif is supposed to be going but it's waves and waves and waves of emails so you installed it it's collecting data but now you're overwhelmed with all these alerts I have an example I went to a client of ours that had bought two network security monitoring sensors and installed them in their control system they're the iciest and one of the vendors out here I won't say who it is because it doesn't matter they didn't tune anything they just took another box put it in and when we got on site to do our work we noticed that basically these things were not tuned there's over 800,000 active security alerts had not been addressed and the baselining feature feature wasn't used for instance there was a mesh radio system that they had and part of the technology with the mesh radio system is to find the best path they'll sometimes change IP addresses and it happens thousands of times per day they didn't tune any of that out they didn't tune any of the MAC addresses there's like only 100 radios that could have put all hundred MAC addresses into the learned database and eliminated all those nuisance along and again bro in Ezekiel alerts on every function code and Richard Balak excellent resource on all things related to network security monitoring he tweeted in November just remind you that bro now Zeke doesn't look for a bad traffic it's inherently policy neutral so you have to craft which alerts are there so collect them all remember beanie babies it's not good to collect them all if you don't have a philosophy so false positives still calls threat alert fatigue this is an article from CSO online and target missed early alert of credit card data breach so they had too much data coming in on the IT sock and they missed the one key alert so we already know that this is an issue with IT your IT sock analysts they have to stay on top of this because they'll miss what happened so there's a couple of articles CSO online and Reuters that talked about this in detail this is a use case for us in ICS security alarm management so just don't collect everything just because you can you have to engineer the process anybody heard about true positive false positive false negative true negative Rubik tweeted this and this a great reference from the developers Google guide on machine learning so false positive false negatives all the learning modes and all the ICS sensors they try to reduce reduce the false negatives and the false positives so if you don't know what it is it's like a wolf is threatened Shepherd said wolf how come Shepherd is a hero so that's a true positive true negative no wolf threatened Shepherd said no wolf everybody's fine that's the two goals that we want false positive no wolf threatened Shepherd cried wolf villagers are angry the shepherd for waking them up and false negative reality a wolf threatened chevre said no wolf and wolf ate all the sheep we have to think about those things as we're building our control systems security monitoring system same way we do with ICS alarms because if there's false negatives and false positives on the control system the operators again they won't trust the system and they will go back into management managing it in manual mode if half if they have to or shut part of the process down so a recap where do we start last year at the on-ramp you have to start small start with what you have and who you already have you don't need to go by all the sensors and put them all in everywhere no start with the key ingress egress points and maybe collect and make note in alerts that are really good for what your process is it's a continuum you use what you have until you need more so crawl walk run and fly don't overwhelm yourself right off the bat and measure what you're doing it's really important to have that sock sock analyst and it's like analyst here go buy donuts for the ICS engineers work together to define the ICS alert philosophy use your existing ICS alarm and sock alert standards and kind of marry them together if you don't have them use is a 18.2 or e mu a naught 1 91 or nist SP 894 as a guide the nist is free the other two or not but they're not expensive start with ICS DMZ firewall and other ingress egress points and create the alerts for those choose existing logs you don't have to go buy any things use what you have leverage your vendors during the process because they're the experts on their equipment say we need your help to tune these things and teach us give us a miniature training or give us a full week training on it also don't put the ICS security alerts in the HMI don't need the operators that trying to do their job to be doing the ICS sock analyst job - so we want to make sure that we're not doing it just because we can shout out to mark Ayala for you know reminding me of that and I agree you know we don't need to overwhelm them on what they already do so well now we're gonna get into play books I'm a Saints fan lifelong and you see everything that coach Sean Payton has every situation every look that offense and defenses has weathered opponents he has a play already cooked up and they practice it so play books and use cases are important for control systems and you have to tie all of these alarms and alerts together in the engineering that you do in your philosophy so have a commodity malware playbook and use case so what will happen if configure Ram it gets on your system what if your credentials are compromised and all the passwords are stolen what are you going to do about it destructive attack there's another one if you have kill disk or destructive not Patriot type malware overriding the firmware and your your serial to Ethernet just like Jason Larson was talking about we have to be able to what do we do about it what's our play that we're going to run coach and then stopping the bleeding you know where do you pull the plugs between your VPN or your connections to the outside world remediation for each play so that play should have a remediation for each one and the alerts should have a tie to each one of these so what do you do restore backups reset passwords etc so run your play design place for each phase practices drills use your player strengths because you know the system better than the attack knows the system or the thread it could be someone made a mistake at five o'clock on Friday that's still a threat how do we get around that the new guy oops he made a mistake let's get back on line fin exploit their weaknesses again you know more than they do about your own network and then finish strong knowledge is the most powerful tool know and harden your network know in tune the network visibility if you don't have enough visibility get more visibility and know what to do in an incident occurs I'll just leave this slide up for those of you who like references a reference the ISA standards there's excellent blog post from Rockwell Automation there's excellent handbook from pas I don't know if you know it maybe they wrote a really great book on alarm management handbook I'm gonna talk to Eddie and see if they can come out with maybe a new book on we can collaborate together all of us in the community create an ICS security alarm and alert standard as well I think we need to do that as a community and here's the slides on ice security alert management so all the NIST and security onion tuning alerts the bro configuration and tuning the developers.googl a page and then applied network security monitoring from chris sanders and then security engineering that's from sierra thank you any questions [Applause] looks like we have a question hi Chris hey great presentation and gots a brilliant idea of rationalizing ICS security alerts using industrial standards now false positives are pretty easy to identify and tune false negatives are a lot more challenging how would you recommend tuning and identifying false negatives in industrial environment well as you know I know you work for one of these companies you have to engineer it you can't just make an alert and say oh let's put it in production no you have to test it run a pcap can you verify that it works and there's a alert on many things or can you make it really specific on some some thing where it's a technical piece of your network or a design in your ear protocol or any of these things you have to think about it that way and test it thank you that's something that certainly the vendors do and we do but from a customer perspective once it's deployed yes what would you recommend it can be difficult to run a pcap through a live ICS yes so what I tell all asset owners you have to have a test system before you just willy-nilly deploy it so do the same thing get a peek at from your live system replay it through the test system and test it out thank you thanks mr. Jake mr. Chris very good presentation thank you question is how do you coordinate between the ICS side that the actual control system side and when they get a security alert for example if somebody's doing a firmware update the security system should start flashing all kinds of interesting alerts but let's say there's nothing scheduled and it starts flashing interesting alerts maybe the operators forgot to talk how do we coordinate these sides so that they talk to each other and you get this kind of collaboration going on because unless during the same room chances are they're not going to hear about it absolutely so that's why I mentioned those teams should work together and they're a long philosophy with the security team you know working with engineers the technicians and the operators and your Incident Response playbook you should have your your all your different players and their contact information listed even the what department heads they are if this happens we have to involve this group of people and it can't be cylinders of excellence you know the right hand not talking to the left hand you have to have that cross-pollination and you know in some big companies that can be problematic and smaller companies like where you came from you were probably both yeah yeah right so it really takes a lot of thought and engineering into it and getting some cross pollenization and that that's why I mentioned the doughnut so don't let's help kolache is if you're in Texas and one follow-up question I talked earlier about internal self-consistency Diagnostics that the operators might want to know about how would you get that information in front of the security people in case there happens to be something coincident going on their side yes so that's where you need to have some trained people on both sides so have that someone that's a sock analyst if you can't dedicate up one person to be the ICS security expert on your sock analyst team get them some training on what they what those processes are and then have a way to look at it there may be a way to take those logs from the SCADA system sending them to the sock and see maybe some basic statistics because a lot of these sims and those tools they'll take logs from any source so we had that and when I worked in the power company we had some some and analytics type data coming into the security system to see also even physical security so if you go out to a substation you're supposed to check in the door alarm should should work and then but if you see someone uploading firmware and the door alarms you know not there and someone hasn't called and checked in we're gonna roll trucks yes thank you thank you and the problem hey I really love the idea of integrating your your sis data into the sim and actually having kind of a hybrid view of it all are there any examples that you could that come to mind readily about correlations between the two like is it common that you'll see a cyber incident that corresponds to CPU utilization or something that affects a particular thing to look for and this is so one of the things you do is take the ICS alarm metrics so how many alarms are they getting per minute just like that graph that shows the alarms coming in you can just like CPU usage you can grab that in but get how many alarms per minute are coming in to to the the ICS network and you could correlate that with these ICS security alerts or the IT security alerts to see if there's anything lining up that would be my suggestion Thanks hey Chris my my company performs alarm rationalization studies and we've been talking about applying the techniques just as you have to to rationalizing Network security alerts one question I have for you you know typically when you do an alarm rationalization one of the first things you do is you get a dump of the master alarm database yes you have all those all the alarms that you can then start rationalizing in this case how would you get a list of all the potential alarms that can come out of network security monitoring tool yes so there's a way to do that and again it depends on the tool and how proprietary it is to the vendor some are more open if you I would actually say work with the vendor of those or if it's more open-source like like bro or Zeke there's ways to pull that list out they have a whole list of all the alerts in their database so like like digital bond they release the some alerts that they wrote as part of a grant and they have a whole list that it's basically printing out the text file and basically having a menu of what'll alerts are available and are they active or not are they disabled right ok thank you problem you said first point is creator and management philosophy what you need for that so I would say you have your subject matter experts that's in the the say the asset owner that is the the lead engineer in the lead on the control systems I would also have someone from a control system vendor that's an engineer or subject matter expert on that not a marketing person or you know sales or you know one of those people actually someone that helps them tune the ICS then I would have some of the key people from the Security Alert I mean sock sock analyst team and you just put them in a room and say don't come out until you get it done we'll buy your pizza thank you thank you sir we appreciate your work - thank you so much thank you so much appreciate it