Help Me With Sign Florida Banking Presentation

Industry sign banking kansas ppt easy

good morning everyone and welcome to the oklahoma environmental training center's somewhat less formal webinar on zoom called hacking your water today we want to thank you for devoting the next hour to this very important topic as presented through rose state college i'm tamara pratt the vice president of external affairs here at the college and i'll be your moderator for the next hour we have wonderful speakers on deck today to share their experiences and knowledge this of course is in response to the water system attack in flora florida about a month ago as many of you already know a hacker remotely accessed the water supply in oldsmar florida and was able to change the amount of sodium hydroxide in the water luckily an employee saw the changes and made corrections immediately no harm came to the public but of course this doesn't diminish the the issue or the need for more awareness and education around the issue our first presenter is ian anderson he is with og e in the capacity of manager of enterprise security ian is a frequent speaker at the ow pcc conference in cyber security issues and today he'll be talking about industrial control systems so we're very excited to have him our second presenter is kevin owens chief technology officer for control cyber incorporated kevin serves as a member on the standards board for the american water works association and he'll be looking at the american water infrastructure act of 2018 and why it matters here in oklahoma to all of us so our chat room is open i hope that you uh as you have thoughts feedback questions you're going to utilize that chat feature and at the end of our presentations we'll be opening it up for q a with both our speakers and utilize your questions from that chat so without further delay let's begin our program with ian anderson of og e ian yeah thank you so much for the the kind introduction tamara um good morning everyone my name is ian anderson i'm a manager of enterprise security at ogne um specifically i focus on cyber security operations and at og e that means um an integrated approach to cyber security so we use it ot cloud all of it all falls within the realm of our enterprise security team and today or tamra asked me to come talk a little bit about the events at oldsmar florida um just a quick you know background on me i spent a few years working for the city of oklahoma city i was involved with the oklahoma city water utility the water side wastewater a little bit transmission distribution all that good stuff and so now in my current capacity at ogne i've gotten to see kind of both both sides of utilities the electric side the heavily regulated side that electric has and then the less regular regulated side that water has now at the events of oldsmar florida you know a lot of us are kind of looking for you know the doom and gloom of the water environment and i'm actually here to tell you um i'm very confident in our water capabilities but as tamara said this it doesn't take away from the need to talk about security especially with industrial control systems and i know a lot of organizations are starting to get more and more interested in defending their industrial control systems and bringing it services and capabilities to bear so i'm going to walk through a presentation that i gave at b-sides boston last year talking about why fancy tools aren't going to save us and really the path to industrial control system security is through relationships so um let me just share my screen all right can everyone see my screen okay i got i got a thumbs up from one of the gyms so i think we're good to go all right so i gave this presentation it's what stickers donuts and listening can do for your ics security program uh our agenda for today is uh we're gonna talk about the problem with tools so right now cyber security is very focused on do you have this tool do you have that tool like we collect tools like their baseball trading cards um and then what do we actually do with them well there's a lot of challenges there we're going to then hit why relationships matter and then we'll follow up with some tips and tricks that we have done over you know that i've done over the course of my career to improve ics security in a measurable tangible way that that provides for long-term success so just a quick disclaimer i'm an i'm a cyber security guy i'm not a process guy uh and i recognize that and i think that's actually turned into one of my superpowers that i can go into a plant i can go into a facility and be comfortable not having a clue what the heck i'm looking at or understanding it right it's an approach that has served me well and i i think it serves others well when you go into someone else's domain and then you realize that you're a tourist and we'll talk a little bit about that so as a cyber security manager here is an eye chart that i'm regularly faced with so this is the list of different vendors that are out there offering tools to solve my security problems but you know the old saying mo tools no problems because you cannot ask any organization even one that is you know fairly well funded like og e or even large organizations to take this eye chart and say okay now here go build a security program from it because now i have to figure out not only what all these individuals organizations do how they differ from each other how they then fit into each other and then how do they actually go solve real tangible business problems that i have right if we are pursuing something beyond checkbox security we cannot just be an organization focus simply on deploying more and more stuff right so there is this little box here on the top left because it's a new category that's kind of starting to get hot these days and that's ics ot now several years ago there was only probably three or four of these companies but now they're starting to grow and so you know some of the big players like uh you know dragos is certainly one of the top in my opinion you have nozomi you have clarity shoot cyberx actually went out and got purchased by microsoft and they microsoft has recently announced that they are folding cyberx's product into the into their iot security offering now iot is very different than ics but we do see this kind of this this pivot by larger vendors to say hey there's this whole world of computing out there that i do not currently have any sort of solution around and so we're seeing these smaller ot companies get bought up and then repurposed for things maybe even outside of the the traditional ot landscape so we go back here and we say wow there's just so much in this you know i grab a new copy of this every single year uh in fact this is now version 2.5 of this eye chart and it gets bigger and bigger and bigger and so the true takeaway in my opinion is there is no tool coming to save us we cannot rely upon security vendors we cannot rely upon security technologies to solve that doesn't mean we can't use them to help us but we cannot rely on upon them to solve our problems so if we're truly going to build a defensible industrial control system environment what do we need to do well first off we need to stop trying to imprint our i.t processes and thoughts into an ot environment because they are very different in a lot of ways in a lot of meaningful ways um we need to actually maybe pivot and say okay instead of trying to make ot look like i.t what what are some of the things about ot that uh make it different that make it more defensible so i've seen this time and time and time again an ot shop you know uh you have plant owners manufacturing lines whatever their job is to produce x widget if you're at og e that could be we're generating electrons and shipping them to your house if you're at a factory it's producing you know a component or a product you know if you're at a refinery it's making gas it's you know those are the processes that actually matter because that's the product the company sells anything that you do to disrupt or hurt that process is going to take away from the value of security so for instance you know we're here to talk about water right if cyber security comes in and hurts the ability for a water treatment plant to produce water and get it into the clear well and then off into the system then security is going to be a threat to the water system itself and if that's how we're being you know good luck getting people to you know to advocate and allow you to come into their environment ot has been around for a long long time 20 you know it's routine to see servers plc's you know rtus all these different devices in an ot environment that have been around for 10 15 20 years i think some of them can now buy alcohol at a lot of plants um and what's great about the ot folks is they have as long a memory as the age of their equipment and what i mean by this is um i was working at the city of oklahoma city and i made a reference to hey like how about we patch this or upgrade this or whatever and the ot guy the operator looked at me and says you know that's pretty brave coming from a group that can't even figure out how to keep my email server up and i'll tell you like that one cuts deep because they're right i mean how many outages has it caused over time and they remember it any sort of outage directly and negatively impacts their service offering plant managers are typically paid by uptime or like they have incentives to to maintain up time so any threat to that is going to you know is going to be a problem um i2 security uh we often like to fill the gaps but ot is different ot it doesn't necessarily um you know how to explain this ot doesn't isn't as extensible or at least not yet because remember we're trying to pour security into environments that have been around for a long long time and so well security kind of likes to nudge itself in between different systems different parts of the environment ot doesn't necessarily lend itself to being as easily malleable as like an enterprise environment and i think a lot about this statement that came from the industrial control system cyber kill chain it's written by uh mike asante and rob lee and mike asante rest in peace he's a wonderful and amazing man did a lot for our community but they wrote this seminal paper about saying what does it take to attack a industrial control system and they they turned the discussion to what are the benefits of ot and this line by understanding the inherent advantages of a well-architected industrial control system network and by understanding adversary attack campaigns against ics security personnel can see how defense is doable what the takeaway what that really means is you use some of the features of ics itself against the attackers ics is very static ics is well known ics is typically monitored 24 7 far more than enterprise environments there's a lot of positive going on in ot but we allow things like the old smart florida's event to say oh you know it's in danger critical infrastructure is in danger and i categorically reject that you know the old smart florida thing yeah i think the i think the biggest danger that can come from the old smart florida event is that we try to simplify security in ot environment and do a little bit of um asset owner shaming because yes was a mistake made sure how long had that mistake existed well where where was i t where was the organization right i mean it all just kind of falls upon the asset owner there at the end of the day and i don't know if that's necessarily fair because the asset owners have very different goals in terms of making sure that their systems stay up so so you notice i tried you know we're trying to you know talk about what what are the different roles that exist here um you know you very much have to approach it secure ot security as a team sport so you need it in there because frankly you know these ot environments are no longer part of isolated or air-gapped environments they are very much a part of the larger ecosystem and environment of you know the organization or at times interconnectedness with other entities so you have to have i.t you have to have ot because again they're the ones that actually know what's going on and why these things matter and how things actually work and then management has to be very involved because that's where resources come from it and ot don't decide how much money you're going to spend on personnel or tools or any of the things that you need to create this defensible environment management does that so if you don't have an active and engaged management that's interested in security you're already behind the power curve so but i want us to really focus more on how do we start to develop relationships with ot because at the end of the day if ot is uncomfortable with you bringing in some of your services capabilities or just your presence alone you're going to have a bad time so what can we do as resource i don't want to say poor but you know resource restricted uh entities that can start to do develop relationships and partnerships that can be leveraged to build more defensible environments well first off is field trips and i know this sounds silly but this is one of the most important things that we do in fact this is probably one of the most negative impacts we've had via coven i cannot send my teams out to the plants now easily and we would monthly go out to the plants if only to say hi if only to drop some stuff off and i've got some other cool things that we do um here in a little bit but god get out of your office go see the folks out at the the facility um when i worked at the city of oklahoma city driving up and down the atoco pipeline i got more information and intelligence from those dudes out in the field than i would have ever gotten sitting downtown so get if you are interested in ot gotta get to where the users are so if you're gonna go out there though be mindful they have day jobs so is the plant in an outage is you know is there something going on is there construction going on are there you know is it a good time be very conscious that when you show up you are a tourist and even if you're no longer a tourist you're certainly a visitor and you create you can't create problems for the ot folks because they kind of have to manage you once you're there so some of these environments are very dangerous and that safety is a uh primary goal of there so making sure that you show up in a way where you're sitting you know the conditions are safe that they have time that you're not creating a burden just by being there so um the reverse works as well so uh we security guys tend to go to some conferences from time to time um there is a amazing ot security conference in miami florida in january and i'll tell you it's called s4 and i highly recommend you going if you you know if you can get the funding and stuff there's so much good information but a pure power move bring one of your ot folks with you so we did this with one of our plant managers and now guess what we got a buddy we got a buddy that's in the plant that can help us understand problems that we see what can help us connect with the right people it was such an easy lift for us to just get one more body uh out there and i'll tell you no one's gonna turn down miami in january um so and it's beyond security like one of the things i really love about ot and our folks out in the field is that you know it's way more than just like a nine to five grind like our linemen are out there like when a storm happens or an event happens our linemen get out there and they they work their butts off and try to try and get people's power back they are really proud and amazing collection of people i'm blessed to get to know them and they do so much stuff beyond just computing and so one of the things that we do is we try to get involved with stuff that they got going on so this image that you have is our annual lyman rodeo it's when all the different linemen come up and they compete against each other and what do we do there we don't pitch security we basically run one of these little booths that you see here on the right we hand out candy and let their kids shoot nerf guns we have our enterprise security banner so they know who we are but we don't we you k ow it's not a security thing this is them this is their event we're just there to support them and so what we do in this is we start to normalize just our presence in a way that you know in a good way like we want to build this positive image attribution not only like they're kind of fun fun events but we participate in things like um uh the uh confined space um uh training and stuff that they do so each plant has a confined space rescue team and they they get together and do training and compete against each other and we're there too in almost identical capacity we're there just to be their friends to hang out to you know be you know just be present again we want them to see our faces and not grown because oh god it's cyber security um so so what are some other things that we do again we try to extend ourselves outside of just uh just cyber security so we hold an annual cyber annual kind of conference to bring some of these people together and we do things like we'll bring the oklahoma highway patrol uh bomb dogs out and then we'll invite them you know we'll invite the guys out and uh ohp will actually blow up some stuff for us and uh you want to get out you want to get field guys interested in what you're doing promise them explosives and they will they will definitely show show out for that so i think this is probably one of our top uh attended functions that we've ever done so and another benefit of this is we have now created a partnership with oklahoma highway patrol so if there's ever any sort of bomb or any sort of problem at one of our facilities it's not the first time these dogs have been around there or the officers and so we feel that we can respond better to a crisis because we've done a little bit of legwork up front okay so now to the really fun part when you go out when you meet with these guys bring donuts bring donuts i mean it is such a simple easy thing a dozen donuts cost you like 10 bucks maybe you bring a box of donuts i promise you people will be happy and you leave these donuts in the break room and you don't ask for anything else you don't say hey i'll give you this if you promise to quit using usb drives no no strings attached you're just there to be their partner but remember donuts are good for break rooms donuts are not good for control rooms or trucks so what we do is we buy big old bags of candy and here's a true pro move start remembering what plant and what operator likes what candy because when someone shows up with something you really like and it makes you feel so special what does that do for your relationship right it sends it through the roof and so we we kind of keep a list of like hey so and so operator is you know is on shift today and we're going out there so we better bring a bag of twizzlers and it's almost like a pavlov's dog reaction when we show up and they see like the candy in hand and they get excited i'm telling you if you have operators that are excited to see you then you're winning because when they're excited to see you they will share all sorts of stuff that's going on and as a security guy that's really good to know i can go fix some things i can go help them i can remove roadblocks there's a lot of power i can do um and also a little kind of fun quick uh story about these puppy dog donuts so we get one of these weird donuts each time we go out and we no none of these linemen or operators want to take the puppy dog donut just because they don't want to get made fun of um but what we do is we bring like a yeti cup or you know some really nice piece of swag we've gotten and we give that to the whoever has the guts to take the puppy dog donut right and so it turns into a reward and a joke and like it just it has this kind of like self-feeding effect you know in terms of your relationship with the entire group so um i'll finish this slide up on swag stuff we all get so if you are an i.t person you know all about swag if you're a security person you really know about swag you go to a conference you're just like they dump it on you guys out of the plants don't get that they don't get the cool security stuff sometimes so we always say just give me a whole box of swag and then we take it out to the plants and like we give it just give it to them right i don't need any more splunk t-shirts i promise but one of the things that um so so we've actually built that philosophy into our deals where we um say hey we're going to buy a product or renew a service have your marketing team throw in a box of swag and just include that in it and they will happily do it like that's that's what they do so you get a box of swag to give out you get get a tool everyone wins okay um kind of starting to wrap up we're big on coins i know a lot of organizations are are big on coins coins are something that we've used uh that og e uses a lot so each year we have a safety coin that we hand out and it's just a reminder in fact i actually have mine right here i always keep it with me so i've got my safety coin it's a reminder of how important safety is everyone has the right to get to go home at the end of the day in the same condition that they they came to work in so we're very very very serious about safety and so we wanted to extend some of that thought uh in cyber security so we started looking at some other coins that we we knew about um you have the beer isac which is always kind of a fun one at the city of oklahoma city we did a coin that we would hand out to people that did a extraordinary job and then at og we also have a coin so these coins are just like these fun little tokens of appreciation when you see something good you can hand a coin and that coin will sit on their desk or in a rack or something and what always happens is someone will say hey where'd you get that coin and then it becomes a competition and then they want to partner as well okay now to probably the main the main part of my uh presentation and i'll try and wrap this up as quickly as i can because i know i'm about about out of time uh talent development is one of the most important and hardest things uh that we do in cyber security so uh make sure that you are looking at your current personnel and finding opportunities to start developing talent both ways take your ot folks train them up on i.t security give them opportunities to learn and expand i t folks if they're interested in ot nurture that right going outside and hiring you know uh third-party ot security stuff is hard because there's just not a ton of people that do it uh the numbers are getting better but we're still a ways away and don't uh don't don't be afraid to deputize people in terms of like giving them access to your security tools if you're a security person it makes them it makes those people better with knowledge um oops little reminders um you know we talked a little bit about stickers and swags um we do mouse pads we do desk mats we don't do it a ton because we just don't have a ton of money for that kind of stuff but we even do little things like we color the uh secure the cables that run our security tools a separate color than everything else because if you go out to one of our plants you'll see a run of ethernet cable but only one of those cables will be green and what that means is that's security and green is that correlates with one of the vendors that we use but it is just like one of those simple little reminders that security is always there and i think that that's that helps with our communication um some pitfalls that i've seen in the past uh one be aware of transactional relationships because your operators just anyone you work with is going to see how empty and vapid those relationships are if you have to have something in return for something then like that's that's purely transactional and you will not develop long-term trust face-to-face is always better than phone calls and phone calls are always better than emails it's hard to do with covid but that is that's the order that you need to focus on get your face time in with these people they're human beings and relationships are developed face to face remember that memories last as long as ics systems do if you break them if you hurt their process they're going to remember and i've been a part of a security team that you know we've had um we've had we've we've performed a scan and it had negative impact on some plcs and we still catch catch hat uh um catch hell for it from time to time so they don't forget they shouldn't forget because this is their livelihood and none of this is a simple function it takes time to earn credibility however the payout is tremendous so first things first they return your calls if you get people returning your calls you're winning they show up to your events you're winning they smile when you show up you're winning these are all the things that we need because what we really need is people to let us know when things get weird we want to be one of the first folks that they call when things get weird when things get hard and by driving the relationship aspect we get that so if we were to try to tie this back to the old smart florida if it had a strong relationship with ot they were likely going to be aware at some level of what kind of remote access technology are you using but since they likely didn't have a relationship and i can't speak for them they probably that's why they probably didn't know because like that's a whole separate world they said well that's ot's problem well no it's all of our problems so drive relationships and uh and i promise you good things will come for your ics security program um so thanks ian thank you so much this has been very a very interesting perspective that i wasn't quite anticipating and i really like it well focusing on the human factor because we have humans running our systems and we can't forget that with all of the high technology that's available to us that it still comes back to people and that that's only built by if i have a french friend if i'm a friend with you and i eat your dog donut bottom line so appreciate it moving on to our next uh presenter is kevin owens i gave you a little brief descriptor of kevin's background i'm not gonna reiterate it again but let's just say he's had a he's been very busy lately you may have already seen him on channel nine most recently and um he certainly has a national presence so kevin i will be quiet and let you begin thanks tamara so uh one of the big things that we're here to listen about is oea america's water infrastructure act of 2018. so there was a big shift in focus to an all hazards approach looking at not just natural disasters like floods hurricanes tornadoes but also looking at physical and cyber attacks so if we look back a little bit with history back in november 1941 hoover saw that water was critical to the nation's economy and how vulnerable it was and he published this in the american water works journal in november and the following month was the attack on pearl harbor the attack in oklahoma city that you probably all remember it back in 1995 that led to a presidential directive focused on protecting critical infrastructure after the attacks in 2001 there was the bioterrorism act of 2002 that came out requiring a vulnerability assessment of drinking water but it was one and done do it and then move on so after the devastation of hurricane katrina then they started to get malware focused on control systems one of those was stuxnet in 2010 and then there was an attack on a water system in the united states in that was reported by verizon in 2016 which led to the updating of the bioterrorism act owea so looking at that update section 2013 of that act takes the vulnerability assessment and moves it to a greater risk and resilience assessment going from terrorism or an intentional act to all hazards in the old bioterrorism act you would submit your vulnerability assessment to the epa now they just want to see that you send a letter that it was done emergency response plan now again they just want to see that you've updated it and submitted a letter to the epa so compliance deadlines we're already halfway through so the larger cities have already accomplished this we're halfway through the medium-sized ones and now it's the smaller cities of which there are a lot uh over 8 000 across the country in this size of 3 300 to 50 000. one of the big changes is this cyber security threat so back in the old days you know it was a very isolated system control systems i was building these over 25 years ago very isolated system but as technology changed people wanted to start having access to that data and so now that that perimeter fence line is no longer the edge once you have that connectivity because that connectivity is going to increase your exposure because everybody wants access to that data they want to see how much power are people using how can i bill for it how much water are people using how can i bill for it so and you look at how quickly a compromise takes a compromise of a computer can usually occur in mere minutes or less here we've got most uh compromises took place um in just minutes and only three percent are discovered as quick as quickly most of them are not discovered for months or and many times that's six months to a year before they actually notice a a presence there are lots of standards and guidance that are out there um awwa creates a lot of these in conjunction with uh ansi so g430 looks at security practices so security practices for your overall company what is the culture how are you taking security as ian was talking how are you taking it is it part of your culture j100 is the risk and resilience assessment and it kind of walks you through those steps g440 has to do with emergency preparedness and these first two are actually in compliance with the dhs safety act which actually protects you for liability insurance if you're attacked by a nation-state actor there's other um guidance that's out there for free all these things you can find on awwwa.org but some guidance on cyber security process control systems as well as a cyber security assessment tool i helped design that it's available on their website for free and you can go through and look at all these things look at connections between your i.t and ot side do they exist how are they protected how are they secured so when you go through this phase one you start looking at where are we at right now so you start digging out your policies and procedures do things like that exist and you start performing that that gap analysis creating an asset inventory is very helpful to understand what are my important critical assets and how can i have a baseline and restore some of these especially after like a ransomware attack so now we're looking at the risk and resilience assessment so some of the things that a utility has to assess is looking at malevolent acts and natural hazards so again that's that all hazards approach we're looking at resilience of the pipes physical barriers source water that's a very important thing water collection and intake pre-treatment treatment storage everything that has to do with it but also the thing that most people don't understand is electronic computers and automated systems and connections to it that gets uh important to be assessed and looked at what are the modes monitoring practices of the system and that's more than just water quality that's also your cyber security your incident detection systems financial infrastructure in the event that happened in 2016 there was a theft of customer data 1.5 million customer accounts their financial data was taken these are people who do automatic payment every month so your name your address your credit card or your banking information all that was grabbed so that's why the oea they're looking at what's the financial infrastructure of the system and many times that's not the water guys domain so that's a separate system in the it side probably in the city system intertwined with a number of other systems it's important to look at that looking at uh storage handling of various chemicals by the system do you have chlorine is it liquid is it gas and um so there's a number of tools that are out there uh again here's a a list of some of the differen standards and some of the associated manuals that go with it as well as a number of other guidances like this i mentioned the cyber security guidance and assessment tool as well as water warn which is the water and wastewater agency response network so the j100 process it's a seven step process of going through looking at your assets what are my critical assets what are the threats against those assets what are the consequences if something were to occur is it going to lead to injuries fatalities is my system going to be offline for a period of time vulnerability analysis so these critical assets how are they being protected what weaknesses exist whether that's in the facilities whether it's in policies or procedures or personnel behavior and then threat analysis here's where we look at what's the likelihood of these events happening you know if you're in uh kansas odds are very low that you're going to have a hurricane so now you don't have to worry about hurricanes and then risk and resilience management is the last step and here's where we're looking at and balancing those costs and understanding is there a net benefit or cost ratio that's where we're doing those calculations now i mentioned source water earlier looking at the source water and seeing if there's events that could infect it are important whether that is a flood or it could be the complete opposite of drought like the oklahoma drought back in 2012. i mentioned warned earlier there's uh agreements that you can make especially if you are a smaller water community system to help if you don't have enough resources if you don't have enough people how can i interconnect or have agreements for people for equipment if an emergency were to occur there's other guidance for a tribal water systems as well as interstates so if you are alongside a border perhaps in oklahoma is there a connection into texas that you have to worry about for instance and so there are a lot of these different standards that i talked about and they all interrelate with each other it's also important to look at physical security what are the access points into that building what are the access points into that equipment that's out in the field what are the approaches and and are there any cameras that are in use or any intrusion detection if somebody opens up a cabinet out in the field looking at the architectural review seeing where those connection points are whether that's remote connections like what happened in oldsmar or whether that's connections into the it enterprise systems so you have the risk and resilience so you've looked at all your risks and you start looking at what are the ways to protect against it and do those things balance out and so in the end you have a report and you look at those vulnerabilities and you figure out these are the priority one ones that i need to fix these at a bare minimum priority two these are going to have significant and immediate increase but i'm not going to do these until i knock off priority one these are going to cost me a little bit more in time resources money priority three now we're looking at i'm really trying to increase uh cyber security in both the enterprise and ics networks and priority four this is if you're protecting against a nation state you will have knocked off all the other priorities here so odds are many of you won't get to this unless you are in one of those critical places where your customer is one of those critical needs phase three we're looking at the emergency response plan so based on what we find in the risk and resilience assessment you need to develop and update that emergency response plan so you want to look at ways that you can improve the security and resilience of that system is there ways that you can detect that event faster or better do you have plans and procedures in case of some of these events that you are brought up in your risk and resilience assessment that need to be addressed they weren't in your plan how can you do that because you're looking at ways to lessen the impact of that event so in the end you have to put a certification letter into the to the epa which has your community water system id number the date that it was certified and whether you were doing a review updating or revision of your risk and resilience assessment or your emergency response plan now i know you're saying well what happens if i don't do these things well under um the utility is subject to enforcement by the epa and for each letter they can find a penalty of five thousand dollars per day both for if you never did the rra or you never did the erp each of those is a twenty five thousand dollar fine per day now again that is up to so looking at those compliance deadlines as i said earlier we are in the midst of those 8 000 plus systems across the country right now if you are a smaller water utility like the numerous rural water systems across oklahoma even though you do not have to submit something to the epa it is still a very very good idea to go through some of these things look at some of these things and address it because if something were to happen you you can't easily look upstream and say hey can you help me out because if you've got problems they most likely have problems as well so that's the wrap up um we've got time now for some some questions sorry i went through that a little fast i was just trying to make it so that we had some uh time at the end kevin while we're waiting for the chat to fill i have a question and this you know the old rule of lawyering you never ask a question you don't know the answer to and i'm openly admitting i don't know the answer to this question but based on your experience i mean this is what you do for a living you help people um with these issues in in the utility sector how prepared is oklahoma from a utility perspective and we're ignoring ian and og e and people like that we're talking about those smaller um smaller rural areas which we know are prevalent in oklahoma um unfortunately uh so i've been looking at these systems for over over 20 years and the systems that i've assessed uh 100 of the water systems have had malware present on it in one form or another the only ones that haven't is the systems were completely isolated or they were brand new and were not turned online yet um but i have seen malware pressing on on way too many systems and you have to understand the smaller water utility systems it's not their fault they they lack a lot of the resources you know for the cities they are one of the the leading sources of income you know because they're putting bills out to everyone every month but then that gets put back into the city coffers and divided up first you know fire and and the police get their chunks and then you've got so you go down and oh we got to give some money to the water and wastewater area so uh you know getting funding for some of these organizations is difficult there are some resources that are available especially if you are trying to buy equipment so if you are trying to purchase like backup generators there's actually funding for that uh the usda is is one resourcer for funding they had over a billion dollars of unused money last fiscal year because no one tapped into it and asked for it yeah doing cyber security assessments there's not a lot of funding for that but there's things that you can do and that's why going through some of those resources many of them that are are freely available go through these things try and check these things on your own well i want to thank you both ian do you have a comment because you're i see you put you popped your face back up so that makes me think you have something to say yeah i know i mean what kevin's talking about it's interesting you know like the big difference between the electric utility space and the water utility space is we are heavily regulated um in in all things and especially around nerc sip so we have a uh you know reliability organization which is nerc the north american um electric reliability corporation then we have our regional entities and all that stuff we have a set of standards the nercsip standards that if we do not comply with we face fines and those fines have real teeth so 2019 duke energy is one of the larger electric utilities in the nation was fined 10 million dollars for having poor cyber security now today and kevin please correct me if i'm wrong but i don't think that there's any sort of like mandated required standards for water systems in terms of cyber security and that being the case like the true challenge is how do you get funding for something that isn't required and i think the other the like the the scarier challenge is what happens when an event creates enough of a political enough political awareness saying hey why don't we make elect water look like electric because they're very different in a lot of very important distinct ways but they'll say well why don't we just make water look like electric and now all of a sudden these other standards are going to come in and so you really don't have the choice right so there is this kind of preemptive like if we take care of our own there's no need for regulatory action however like at what point does that become a real risk and i think that these are these are some of the modern problems that uh water utility managers have to grapple with and saying do i am i going to be that final that first domino that kicks off you know a true regulatory effort in that space and you know to introduce required standards i don't know kevin do you have any thoughts on that well um the the the standards that are out there like i was talking about the awa ones uh they are not required they are recommended but not required yeah so um i'm currently updating the j100 right now trying to put more things in there focused on cyber security as well as ways for companies to easily assess these things and actually put numbers to it because if you're looking at if i get hit with ransomware and the system gets taken offline what is it going to cost me to recover and so you put a price tag on it even if it's a small water utility or a small organization you put a price tag on out of like a hundred thousand dollars now ian you know that a hundred thousand dollars for a ransomware ransomware recovery is is nothing i mean odds are it's going to be much higher but you put a number 100 000 in there and you say or you can create a backup of that system and have a plan for how to restore that yeah and you could put something in place work with your it guys yeah and if you have to hire someone outside it might cost you five grand and i had a city go kevin it wasn't five grand it cost us thirty five thousand dollars i got thirty five thousand dollars what did you do well we're completely upgrading our server and all of our engineering laptops and all of our computers and that's why it's costing us 35 000 and i go okay 35 000 versus removing a hundred thousand dollars in risk annual risk and they go oh yeah it's still still a great deal well i mean it and it gets down to there's there's i i thought your financial infrastructure discussion was really interesting because i i think that this is something that all sides of utilities face so the the capital constraints and o m constraints that um utilities face so if it has to do with like subscriptions personnel the day-to-day stuff the o m that goes directly onto customer bill so what what i'll tell you is that og e we are incredibly sensitive to the customer bill right and we have very clear you know guidance from leadership that we need to protect the customer bill it's really important for us to keep that low and so we're challenged to find solutions that don't necessarily have you know negative o m impact and so but things like rape cases things like uh you know you know i know city council for the city of oklahoma city approved a rate hike a few years ago probably four or five years ago and that money went direct to infrastructure upgrades cyber security upgrades like what utilities really need is support and i think like reach out to your corporation commission reach out to your city council say hey this stuff is important um you know the city of norman had a a breach not necessarily of their control system but of their uh billing system and so customer utility billing was exposed and frankly it's that announcement was made and then it went into a black hole right so no one's talking about it anymore so i what what is required is continuous sunlight on these problems and forcing leaders to discuss and say okay like you know how are we going to defend against this how are we going to recover because at the end of the day politicians and city management still have to make this decision of like okay do i add another public safety officer or do i spend this money on a blinking box or doing something that may or may not happen you know politically speaking it's always good to add public public safety and i know our public safety departments really need additional manpower as well so what are the like that's the nice thing i like about required standards is you don't really have a choice at that point well the reason why they most likely will never be required is if you're looking at nerc sip yeah you get an outage it affects other communities around you they can't okay you take a look at the the you know the cascading blackout around the great lakes um when you look at water you are affecting only one small water system what happens there isn't going to happen on another water system that's right next door yeah that's right well and actually if you really look at nerc nerc only applies to what's called the bulk electric system it does not refer to the distribution management system so this the grid that connects to your house is actually not covered by the regulatory authority of nerc it's just the big tall transmission lines and the big large power plants that that represents and we can get into the nuance of that but no i agree right like water is far more a distributed resource than um you know if we're comparing the utilities gentlemen i'd like to jump in here i'm sorry it's a great dialogue but we've got one comment on the chat and one question that i want to get to polly edwards wrote that homeland security uh grant cycle has just opened it's a huge initiative on cyber security local or small systems would have to apply through the state so that's great information thank you paulie yeah there's 1.9 billion dollars that just got released so that is an open cycle right now and if you are in a rural area or a tribal area odds are significantly higher that you will get the funding that you're looking for including for cyber security thank you kevin that's that's good information and this next question is for you um also well let me follow it up stephanie asks what state agency do they apply through for that homeland security grant uh then i'm not sure i will i will have to find out and i'll i'll let the tamara know okay thank you we'll get back with you paulie says oklahoma homeland security probable i would assume it would be a oklahoma office of homeland security but let's you don't want to make an assumption right you know how that goes so the question the final question that we have for today is homeland um how prevalent are the remote access systems that were compromised at oldsmar and how do you typically recommend that organizations protect such systems kevin so at oldsmar um the credentials was one thing that was compromised so there's a website that's out there called www.uh.haveibinpone.com and so the credentials for that person their username and password was for sale on the dark web and had been for some time so that's what was used in this particular incident so routinely changing your password you know you hear it say it hey that's a good thing to do you know every 90 days at least when it comes to the ot side i have seen people go i've got my same password that i've had for 20 years and i go what is it he goes enter and i go the word enter and he goes no just push the enter key it's null and i was like wow okay um so you know what's crazy is that's actually more secure than some of the other passwords because no hash c ackers are looking for a null entry okay but is that good is that good practice though no of course it's an interesting component of the the cracking but yeah and oldsmar also had all their systems were running windows 7 which ian could probably tell you is is not the best thing to continue running because it's out of date but again it's resources having those resources the guy who was i t for the water sector was also i.t for the the uh wastewater sector was also i.t for the city but his day job was be the city manager okay so all those i t skills were just side little things um and one way to secure those systems if you are doing remote access is you look at multi-factor authentication there are ways out there and there are companies that produce a simple solution for five dollars per month per user for multi-factor authentication that's if you go sign in your username and your password boom it puts a pin to your smartphone and say yes allow this access if they would have had that their system would have been secured thank you kevin thank you ian so much for your time today we are out of time we're right at 11 o'clock i want to be respectful of everyone's one hour time frame thank you again so much everyone thanks to our presenters i'm going to give a virtual applause it was a very interesting dialogue and i really appreciate it we'll we'll be doing more of this in the future um thank you for your time on behalf of the oklahoma environmental training center we appreciate you being here and hope you have a fantastic remainder of your day thank you everyone thanks have a wonderful day be safe