Streamline Your Document Signing Process with Pipeline SCADA Alarm Management for Inventory

How to create outlook signature

thank you thank you all right thanks for coming and uh thanks for staying awake after that wonderful lunch uh I'm going to try not to talk like Ben Stein this is a pcap well I'm on it says I'm on do I need to speak louder okay now there we go all right let me start over thanks for coming and uh thanks for being here after lunch that was a wonderful uh barbecue lunch right and how everybody's gonna do napcon right now so especially if I start talking like Ben Stein you know this is a p camp and it's really comfortable and it's really no I'm not going to put you all to sleep uh what I'm going to talk about today is tuning ICS security alerts an alarm management approach my name is Chris sistron technical manager with ICS team at mandiant and I've been here almost uh nine years so and now Mandy it was bought recently by Google so I'm going to be getting one of these hats pretty soon I think I'm going to try to put the propeller on my hard hat um before before Mandy and I worked for a power company in the South called Entergy I worked there as a as a power engineer skate engineer for almost 12 years I founded and ran besides Jackson in Mississippi for about six years co-founded the beer eye sack which is uh the best ice act and we're going to have beer Isaac later you know after the conference everybody goes and does that you know get together and the founder of napcon and dad jokes as a service uh quick uh I love the the dad jokes earlier uh um I'll tell one dad joke and then we'll go uh keep going uh how did the criminal cyber criminal escape the police he ran somewhere all right so so with this uh you know you know got some Mandy and alumni in the front row over here and over here we got some Indian alumni uh what's going to happen is it's like we became FireEye and then nope it's always been Mandy now so all right so about the talk uh threats and risks aren't going away so they're going to guide and direct our detection and response goals you all know this how many of you here have control systems at your company or in your day job a few hands okay you probably have them anyway just to keep this in mind if you have any building automation HVAC like data center AC you know for your data center we found those on the internet you know um you have control systems all around you so this is something that to keep that in mind or if you have customers or clients that have control systems this is going to help you because this is still an emerging field even though I talked about network security monitoring back in security onion conference back in 2015 um it's still relatively new there's a lot of control systems out there that aren't monitored and so this is kind of a talk for those who or more advanced who have some monitoring capabilities may be bought one of the newer control system security network security monitoring products and put it in and now what do I do with it so the whole deal is is we need to have a way to engineer the system and have a philosophy around how we tune the security alerts that's coming in and so I like to call this security alert engineering or as Chris and folks here call it detection engineering right um Josh yeah and uh we want to call this is very similar to how control system alarms are tuned and we're going to talk about two standards that exist out there this may not make any sense to you but this is we're not Reinventing the wheel here tuning is not new by any means and if you talk to control systems engineer they'll go oh yeah I know what tuning is we have to do that whenever we commission a new control system and then we'll tie that back to like the nist standards and security onion best practices of tuning your system and then lastly we'll talk about response and a little bit about playbooks and such so there's some really great content around Control Systems security engineering it's a emerging field even for our group so there's some OT security blueprints from Sarah flukes from Germany and a couple years ago at S4 is one of the probably the the control system security conference to to go to there's this birth of secure coding practices for plcs most of the time they don't have any security or if they do it's not enabled by default so there's a lot of cool things that that whole team has come up with Jake Brodsky Sarah Vivek Dale Peterson and more and then recently also born out of S4 was this idea of incident command system ICS which is like Nema if you have any Nema or incident command folks we're following that same process but for doing incident response and control system environments called ics for ICS so that's a little plug about the stuff that we're working on in control system security basically the Crux is know your systems and you all know this if you don't know your system you can't defend it how does my system work what are my threats and risk do I have enough visibility do I practice my plans and we've had a lot of great talks today about all of these things especially at visibility and how how things work how could it be used against us how do we find evil so here's the problem statement there's little published about Control Systems security alert management asset owners have to learn things by doing that the hard way and there's really no guide about it even still even the manuals for these network security monitoring sensors don't have a whole lot in their manuals about how to tune the system so the theory is ICS alarm management like for like a Siemens system or an Emerson or a GE or whoever those systems have very well defined tuning standards and there's even standards from the international Society of automation that's well defined and it's been talked about a long time it security alert management is pretty well defined I mean I remember back in 2015 there was that talk that talked about collating alerts where you don't get the same alert 50 times you just get one alert and then you can just open it up and see the 50 instances of that alert that's been well defined now we have to say I see as a security alert management must be engineered just like we engineer the control system we need to engineer the detection system so the solution is to create a reference that combines both Concepts to empower our security teams and asset owners so I like to talk about history you could probably get me talking about you know old you know glass insulators or old power lines you know the way things that were done serial ports and modems old laptops dial up you know all that stuff we've been doing this for a long time ICS network security monitoring is kind of the new hotness like Tesla is that everybody's got oh we got to get a electric vehicle um that's going to be great I saw the plug-ins in the the garage across the street this is really great um but it's not new um here's a page there's a whole 50 page section of the in standard handbook for electrical engineers and that's the fifth edition from 1922 that's a tweet from way long time ago they have a whole section on electric vehicles and then they have a whole nother section on electric uh street cars and electric trains and locomotives and there's a picture of one if you go in like Jay Leno's Garage he's got an electric vehicle you know that was from like 1915 or something like that so what was all this new again see where I'm going with this Engineers already know about how to do monitoring and I say control system Engineers electrical engineers mechanical engineers and all these this is something that's not new we've been monitoring processes to get an idea of how it the system works and how to improve it and what the root causes if something goes wrong so on the left there is a what we call a wig wag chart that's the nickname it's a recording meter and it has a little pin and it records something like volts or megawatts or something at a power substation over time but it can monitor temperature pressure anything like that the one on the right is down into the a-phase power for inside of a relay when there's a fault on a transmission line you can actually see when the voltage collapses and then when the current increases for just a few Cycles there and you can see the the phasor diagram of what's going on so we've been doing this kind of root cause analysis for quite a while here's the timeline go back all the way to 1839 Charles Babbage invented that chart recorder we talked about and doing monitoring on train Railways I uh they they headed on the back of a rail behind a locomotive we fast forward to what I'd like to talk about power grid that's my background of course three phase AC was in conceived around 1880s in 1888 the Bristol chart recorder was invented just for the power system and that's the picture of the patent there on the left first scada systems were two wire telephone based and they had contacts to see when something was open or closed and then they could send a telephone relay to open or close think of like closing a breaker opening a breaker or like even on railroads they would do a switch change the switch on a track and that's a picture of one of those ones what it looks like if you fast forward to the 60s we have a lot more computers being there we have digital scada coming online and then we have the Northeast blackout boom something bad happens okay that really causes a revolution and when something bad happens safety rules are Written in Blood so now we have to go write some safety rules we have to invent some new technology to prevent the Northeast blackout from happening again so we come up with do fault recorders new digital relays we have ethernet based scada phasor measurement units things like that and then I have in the bottom right we have network security monitoring it came around in 1988 you know the Cuckoo's egg all that stuff we all know and love you know so then we have the blackout happened again in 2003. and we go well we didn't have enough forensics data this wig wag charts that show amps was not you know detailed enough we needed more information we couldn't see down to the milliseconds on the forensics to put together what happened they did put together what happened but one that after action was we need to do more and so we go back to this and you can see down to the individual cycles of the sine wave that's on your power lines and then now we've fast forward uh where we have the Ukraine grid attacks in 2015 and 2016 and and then the Ukraine Russia war against Ukraine now uh we see that and this kind of just gives you an idea this is just engineering for Power Systems this is just an idea of the power grid monitoring since then so what I try to do is if I meet someone who doesn't know about security monitoring I go yeah you do you just call it something else you're doing power grid monitoring or you're doing control system monitoring so that we're going to see where I'm going with this we're going to take two concepts and they're basically the same thing so a little Bobby I don't know if you've seen Rob Lee's uh comic strip let's get some Network visibility and more secure reliable safe environment what are your goals great and you might want to think about to the in-state deliverables with the scenarios and impact those goals and think about the in-state and work backwards make sure your response scenarios are supported by your detection and collection strategies so that's a really great little comic strip around network security monitoring and control systems so it's really great so let's take this standard um Isa the international Society of automation has 18.2 2016. this is really nerdy and I will read just this first paragraph the primary function within the alarm system is to notify operators of a normal process conditions or equipment malfunctions and support the response does that sound like what we do I mean this is pretty much the same thing so let's read nist intrusion detection is a process of monitoring the events blah blah blah of imminent threats and violation of computer security policies acceptable use policy standard and basically ends up supporting what the response so we take this diagram of a control system and it shows from this standard this Isa standard shows where you collect data from and where the alarms are going and you take these different sensors on the farther the far left you have like a pipe there it's got sensors it's like temperature pressure monitoring there's a control valve that has the safety system that's the sis or the basic process control system that's the plcs and you have your hmis that you touch and can turn on and off things and all these things are collecting to an alarm system well let's drop Security on top of that we can collect the alarm logs and the security logs like the newer plcs have syslog in them and like I talked about in 2015. we can collect all those logs and even collect them from a network security monitoring system on a sensor on the network and send them to your security system your sim or your sock or whatever it could be security Union basically it's taking the best of both worlds of operations we're monitoring the process we're trying to make sure that we're still making power we're still making oil and gas into end products that we can use every day or maybe chocolate bars it could be anything right and then you have security monitoring the network for malicious activity for all these other things like regulatory safety things basically it boils down to with visibility like talk earlier you can't see where you aren't looking and neither you you can't do forensics either one thing that I run into is engineers in the control system environment don't really know the term dfir or forensics they call it a different term they call it root cause analysis so sometimes this is boils down to a conversation you need to have around terms so root cause analysis they understand that is the same in our world as digital forensics there are engineering forensics like after the Super Bowl blackout and uh you know after after in the Superdome they had a forensics engineer come in to determine what happened and they cause a root determine the root cause I got a phone call because I worked at that power company and they said is it cyber I go I don't know yet I just saw it and let me go log in uh it wasn't cyber thankfully uh that actually that substation had 1200 baud scada modems so I don't think a hacker is going to go waste his time trying to fool with that uh fortunately it was not related to uh it was a defect in the relay that was there protecting it it wasn't anything related to cyber security at all so I love this uh Josh uh and Chris talking about detection engineering is a process of researching threats and then building and tuning the tools that find them I love that I love it love it I freaking love it you know laughs uh because it's talking my language it's talking about when we don't willy-nilly just go buy this and throw it in there and then yep yep we checked that box nope uh we want to get rid of these nuisance things we want to custom craft custom build uh and this detection system around the thing that we've already custom built which is the control system so this alert philosophy in the ISA 18.2 standard basically it starts with having the basic definitions that extends to operational definitions and then you have the performance and metrics around that based on the objectives and principles for the Alarm Systems this is very what we do in cyber security with a mature sock operation you're going to have the same type of philosophy around it so what I would say is let's have an ICS or control system security alert philosophy around defining the security operations defining the alert categories and priorities defining the metrics around them and then align them with your existing philosophies that are maybe with your it alerts security alerts or your ICS control system alarms so it would be cool to have like your playbooks that have not you know the the university folks say what do we do I would just Google it no we want to have a known custom response for this event just like there is for if a temperature alarm goes off on a critical pipeline you know you don't want to have to go I got to go read the manual no we want to have it Custom Engineered the entire way there's a a free checklist in addition to the ISA standards there's this one called eemua based in the UK engineering equipment and materials users Association they have a standard called engineering eemua 191 basically talks about how to do a checklist and for your philosophy creating these alerts so we can use this to create ICS tuning alert philosophies so just replace the alarm word with alert and it's like Define any technical terms used in instead of eclipsing suppression we just change change it to ioc network attack sand worm you know whatever custom alert you're riding it for change it you know lay out the methodology for managing the alert systems including the management of Change setting in the alert priorities or incident priorities the guidance on the selection of the rules and things like that so basically we're taking something that's old and making it the new hotness we're doing the same thing that we've done before a trillion times we just forget in human history that we've already invented it same thing has this whole philosophy around security alert management this audit philosophy Loop that's in Isa standard 18.2 talks about how to have a mature alarm philosophy well let's just change some of the things from operation maintenance and monitoring assessment to security Ops tuning and hunting it just Falls right into place I love it so going back to the old mandiant presentation I gave back in 2015. how do we do NSM for Control Systems there's a little example of control system you know grabbing the firewall logs or you netflow or full pcap P CAP or didn't happen and then detection looking at your security onion alerts or your firewall alerts or high CPU usage or whatever we want to consider where we detect and then apply it and tune it so this is some diagrams from alarm management there's a blog post after this I'm going to post these so you can get to the links but it tells you what happens if you have not tuned your system so basically the nuisance alarm should become standing alarms these standing alarms become performance Target missed alarms then you're getting into like wear and tear of Machinery you're getting equipment damage and if you are basically upset so much where you're overwhelmed with so many alarms you don't know what to do with you're missing critical alarms you're going to have injuries and possibly deaths emergency shutdown or environmental violations if you can't manage your security alerts the same way you manage control system Engineers manage the alarm system so you see we can use the same philosophy as well in the aamua blog they have this level average number of alarms per operator per hour and then maximum number of alarms per operator per hour and there's a a specific level just between level four and level five is that their target what's uh and we have this for socks right we have how many alarms or alerts can or incidents can a sock analysts handle how many can the hole saw handle it's the same thing with control systems we don't want to overload our people and if you only have one person and you have 6 000 alerts per hour then that's that's untenable so how do we reduce nuisance alarms and reduce nuisance control system security alerts just the way same way we try to reduce them with like security onion tuning right so here's an example before from this blog post from chemical processing.com that had a reduction of nuisance alarms they have the process of alarms which is up in red and the operator's actions which is blue below once they reduce the nuisance alerts you can see that the the signal and noise ratio is changed significantly an operator can see when they need to act upon which alarms that are happening same way if you have too many cyber security alarms coming in right and there's another one about this blog post about fire alarm systems because you know someone pulls a fire alarm okay is it you know is it a bad sensor you should clean the systems uh regularly avoid activities that would trigger the alarm or like propping the door open and you know the oh there's a bakery next door that's got too much smoke happening you know or maybe you're just using your smoke alarm as a timer in your house when you're cooking right I've done that seasoning cast iron at 11 PM one time my wife was yelling at me and I realized that I should have propped the back door open and turned on a fan to prevent that from happening but it still it was I barely made it out of there with my uh don't wake your wife up with a fire alarm okay so examples when you don't tune you know this your favorite security engine sensor or your favorite control system NSM sensor here installed its collecting but data oh my gosh you got all these emails all these vehicle alerts coming in oh my god um I have a little War Story we had um a customer uh uh that we had the head did this they went out and bought a you know a latest vendor I won't say which vendor um and it doesn't matter really they just one for their water system one for their Wastewater system and they weren't tuned at all and they weren't trained on how to tune it and one had a million alerts in it the other one I had eight hundred thousand alerts in it the one that had a million alerts in it you had to reboot the sensor to be able to log into it so just imagine if your security onion box Doug like you know it's got a million alerts in there and it's not tuned it's it's useful it's useless what would it happen the root cause was is they had mesh radios that change IP addresses based upon the best route and so there was all these alarms of uh Mac address IP address changes if we would have just tuned to learn all the known radio Mac addresses we wouldn't have gotten any alarms and you know our uh our good friend Richard balek tweeted about this it's like by default bro or Zeke is going to generate alert for everything so if you oh we uh experienced this if you've ever put Zeke into place you've never put it before and you by default it's going to alert on everything so modbus or dmp3 it's going to alert on everything so you got to turn off everything and and then turn on things that you only want it to do so it's uh it's it's a good thing to get rid of that and you know collecting them all everybody remember Beanie Babies um yeah that was something right back in the early 2000s right um they're not worth anything if you collect them all you want to collect the alerts that matter you know the I wish I still had that bow nose baseball card right I had all the baseball cards but the one that Beau knows that had the football pads and that's the one you wanted right as when I was collecting cards as a kid false positives still cause threat alert fatigue and like with the target breach they missed the critical alert because they had too many alerts in there there's a couple of examples from the news um uh if you haven't seen this before this was really great this is from Google uh and I didn't know that I was going to become Google so but this is from a few years ago when I found this webpage they have the confusing Matrix of true positive true negative versus false positive and false negative example of true positive a wolf threatened Shepherd said wolf how come Shepherd is a hero false negative a wolf threatened Shepherd said no wolf how come the wolf ate all the Sheep and and then uh everything there mubix on Twitter gave me the idea to find that so shout out to him for that so where do we start for those of you who have control systems or have customers clients friends whatever uh if they're starting to do security monitoring in control systems where do you start is generally there's not any sensors at all not very logs being collected unless it's for critical things so basically we want you to start small and use what you already have you don't need to go buy the latest and greatest everything and redo all of your control systems with the latest Windows the latest Linux the latest plcs no start with what you have and then use that capability until you need more right don't overwhelm yourself right off the bat and then as you get better you add more add more and add more I gave it a little bit more details about this in my talk at s419 and that video is on YouTube so focusing on the basics um sock analysts go buy engineers and your technicians go buy them a box of donuts and ask them their pain and they will tell you what has been painful around I.T and I.T security and then go solve and Conquer because they're smart they know about monitoring they just don't know your terms for monitoring remember they don't know DFI or they know root cause analysis RCA um work on this alert philosophy together and leverage your existing standards like your alarm standards or your sock Philosophy for creating alerts if you don't have one you can use these standards that I mentioned before start if you need to know where to put a sensor start at the boundary between your ite and your control system or OT it or OT sometimes there's a DMZ firewall sometimes there's a switch it could be somewhere but there's always a delineation between the two generally even if it's a flat Network there's still a delineation there and then add your existing Windows locks there's a lot of Windows systems are in control systems and so they have Windows alerts you know applications security logs all those things the one piece of advice I have is don't put security alerts on The Operators workstations they already have enough to do managing their own system so leave that to the stock analyst to do that don't want to get unless it's like one and like if security alert any tell them to call the sock team maybe that's the only one but generally try to keep that away from from what they have to do all right now we're going to talk about football and play books and uh this is a really great picture of uh Sean Payton who used to coach the Saints and you could zoom in and see his Playbook here great thing about football is you have a playbook for everything right for third down and two or fourth and long or do we pun do we pass or do we run right um the the great thing is about the coach is going to use the strengths of his players against the weaknesses of the opponent and that's what we want to do as Defenders so for it we already have tons of playbooks and y'all got even playbooks in security Union which I love this is great but we don't have that many in control system environments so let's think of four basic playbooks that people should work on for control system environments one commodity malware and OT what do you do when you have conflict or Remnant kegertip any of these regular old windows things that's been floating around in your network for since 2015. well there's generally it's not hurting the control system but it could and so you need to have a way to find it hunt for the rest of them that could be there and then clean them safely I've helped many many customers do this and it's a good playbook to have um okay the second one is OT credential compromise a lot of times um Control Systems have problems like you can't change some passwords there may be hard-coded or they may have less complex abilities like you can't change it to 20 characters and have all the special characters um maybe they're only limited to eight so you need to have that knowledge beforehand in case your control system credentials are compromised like what happened during the Ukraine power grid attacks in 2015 they stole all the operator passwords and did remote access there was no a single factor and what was happening is they were turning off power they would notice it and then kick them out kill that one operator account and they were playing whack-a-mole and they stole the next one they stole the next one they just had all they had all the accounts instead of just severing the network of the VPN or severing the it and OT Network to cancel all that out they didn't have a Playbook to to what to do for or when their credentials were compromised a destructive attack in OT uh this has happened like kill disc with Ukraine or uh not Patia or want to cry you need to have a red button sever emergency isolation event for segmenting your OT um and there's lots of different ways to do that lots of philosophies on how to do that it's really important and obviously black backups but uh and then the last one is ICS protocol attack we've seen more and more uh two this year happened um in Destroyer 2 was a new uh version of industrial one from the Ukraine 2016 attack thankfully the uh Ukraine cert caught it before it was deployed but it was targeting their IEC 104 protocol and then there's another one called in controller that we wrote a report on Andreas wrote a report on as well [Music] that we caught or someone caught before it was deployed in a victim network but we saw we actually ran it in the network with Schneider Electric one of the vendors that was targeted uh their products were targeted in that so we worked with them and ran it in the lab and it actually works so someone what do you do when someone's fuzzing my bus or sending unsolicited uh dmp3 commands or doing function code 90 and modbus to reboot your plcs which is a Metasploit module by the way what do you do in those cases so have have a playbook for all of those uh for if you can and since um since we're close to Atlanta uh I Gotta Throw in that uh gift there of uh Tuttle uh making a quarterback Matty Ice hit the hit the Hit the Dirt um the design plays for each of these phases practices drills user player strengths exploit their weaknesses and finish strong that's my thing here I think security engine should have a mascot because ogres have layers as well so um so knowing Harden your network knowing tune the network visibility and know what to do when an incident occurs Doug I think this I don't know if you can use this one it's copyrighted but uh but you should okay well ogres have layers cakes have layers so here's a eye chart for all of the references that I talked about this will be posted or you can take a picture and then uh so this is ICS ICS alarm management for all the stuff that Engineers have been doing since the dawn of 1880s um ICS Security will want to also leverage the security alert management so got all the stuff from security onion in there and this 894. um Zeke developers Google and like apply network security monitoring from Chris Sanders and Jason Smith and then the security engineering blog there so I want to appreciate everybody for attending my talk and I think I'm the last speaker before this a break or the state of the union or something like that so do we have any questions yeah oh yeah well those aren't it's like they don't there's no logs needed either if Sea Squirrel assume it was the cause and then we would have stories of uh line Crews if they couldn't find out the cause they would take a dead Square out of the back of the truck and throw it on the ground and say there's the culprit that was called a ground squirrel or a throw down squirrel um so yeah squirrels is the normal uh well number three calls of power outages any other any questions so go home go back to work find your control system environments including your HVAC your elevators escalators all those things you'll want to put some monitoring on those and if any anybody makes a change through the rooftop unit for your AC you want to know about it right so anyway thanks [Applause]