Deal pipeline software for Security

How to create outlook signature

well welcome thank you for coming here than the first talk of the day we are going to talk about building a secure pipeline I really meant to put the word app SEC in front of pipelines so that's why I have the print parenthetical phrase EFT underneath it that was my bad because this is really about how to do taking the DevOps and agile and the things that have worked really well for Devin in some cases infrastructure to give you doing infrastructure is code those things that worked well for those groups and making them work well for app SEC teams so a long time security and be careful hanging out with them because you'll start roasting because I did after I met him and now I'm a coffee snob it's it's it's a good and a bad thing and I'm at two sorrow and like Aaron was saying we did this together at Pearson several years ago for the first sort of iteration of this is where the idea came from and I really think it really really worked for us and I really think the industry should look at this as the sorts of ideas in a way to make them there are jobs as absolute professionals much better and we started the app set pipeline project rather at a wasp because we realized there's a bunch of good stuff at a loss but there really wasn't something for internal people running teams and so we're trying to make the app sick pipeline program be or project be a way that people that are running an app sect team or maybe you're a team of one which has happen to all of us help make your life a little bit better by gathering resources and bringing things together so I want to talk about the evolution of app sack and how I think we can evolve from where we are now to where I believe we need to be and I want to start by talking about trains which kind of seems weird although there is this whole MIT Lincoln lab and train thing there's this weird thing with the early computer people in trains I'm not into trains personally but I want to talk about trains because trains besides the whole steam engine revolution trains particularly in the u.s. really made a radical change right we went from being kind of on the East Coast where the Brits and etc landed and poked into the the new world air quotes right but when we got trains we suddenly can go from east to west coast and America became kind of one big country because of trains right so if you look at trains right travel time before a train was six months and then a wagon and you know maybe full of Indian arrows who knows right but it was a pretty arduous journey you didn't lightly decide to go across the US that was kind of like we're making this bold move and we're gonna strike out right I'm going to West and it was it was an adventure it was exciting it was potentially dangerous after we had the International inter-continental railroad that went to a week and a week is not much time in the 1800's that's that's a silly fast and it was 150 versus a thousand eighteen hundred dollars right that is an amazing drop in price it radically changed how travel happened in the US and if you didn't by the way have a train stop by you more than likely if you didn't have a train stop at your town more than likely you're Thailand I doubt or it certainly was hurt and didn't couldn't perform as well economically without the train going by there were many towns that the Train decided to not stop at and they just kind of withered away and died so let's talk about trains and change right it changed the landscape better or worse I'll let the historians argue about that I'm not a historian I'm a geek but definitely the u.s. got smaller right this big huge continent that was six months in scary and wagon wheels and no roads to get across suddenly became a one-week journey and a reasonable amount of money right that was a huge revolutionary change and it's valid markets right if you made something on the East Coast you could ship it reasonably to the west coast particularly it wasn't a perishable good right this became very easy to do before you had to put it on a wagon and had to make it all the way six months later who knows right and certainly the cost of going west went way down from a thousand to one hundred and fifty that was a huge radical change let's look at this DevOps I think changed as changing or it has changed IT for better or worse certainly small batch size and change size got smaller with CI CD fundamental principle of DevOps we have increased agility and more customers because now we can do those experiments and they're cheap and the cost of experiments goes way down and you can write code push it to the github and it goes into a production or pre fraud depending on how you want to work things your cost of experimentation goes way down it's not a six month rollout with a change control window and gating and all the other hoo-ha that traditionally happened in software right this is the same kind of speed up that app sect needs so I'm really waiting to see when app sec will give the love to DevOps and bring some of these principles in because I think this needs to happen we need to meet in the middle somewhere like kind of like the Transcontinental was at promontory point I think where they put in the final stake of the Transcontinental Railroad and luckily they weren't off that would that would have been really hilarious suddenly this big curve right before the end but we really need to get together and meet have a meeting of the minds so let's talk about pipelines and particularly apps like pipelines and I've had some confusion why I've given these talks before where people come in thinking I'm talking about you have Z ICD how do you add security to it and that's cool but that's honestly not where we started we started with let's make a pipeline and that pipeline delivery isn't the artifact or the output of that pipeline isn't a deployment or a built piece of code it's security artifacts right it's results it's reports its bugs submitted to a project or an application or what have you right so we're making a pipeline not to build or deploy code or making a pipeline to build their deploy the output of a security team ie what's the security state of this our suite of applications so let's talk about knapsack pipelines and like I said I completely stole the idea of pipelines from CI CD because I think it just works and what what is it if you're going to do a knapsack pipeline what are the key features what are the things that make it unique you have to design it for iterative improvement because problem most likely none of the people in this room were hired to be pipeline builders you were hired to be app psych professionals right so you have to sort of fit this around the day job in essence carve out some time to do this and you're never gonna have that hey I need to go idle for 6 weeks and build this thing right you're gonna have to build it in pieces over time the nice thing is once you get the set there's a reusable path for app second you start to have a common language amongst the team and amongst your constituencies about how things are going and where they are in the phase of a request coming in to results going out and there's the one-way flow with well-defined states like your you've put in a request I've had your request now you're going to D it Dastur SAS testing okay your results are in the vulnerability database now here's your reporting in JIRA or whatever bug tracker you want etc right we have well-defined States as they flow from left to right and the probably the biggest thing is it has to gracefully interconnect with whatever your developers are doing in terms of how they handle bugs right if if you're not putting security findings into whatever bug tracker your developers use you might as well be speaking French to people who only speak English right like they don't want to see a 300 page PDF I want three hundred page PDFs to die and go to bed forever I never want to see him again I don't want to produce so I've produced hundreds of them and it's awful so I'm a reformed PDF producer but it really does need to gracefully interconnect with it with the dev teams so let's look at a pipeline so when Erin and I were at Pierson and we were trying to figure out our situation that was what it was how many developers do we have it was like twelve fifteen hundred twelve fifteen other developers so there was eight of us do it right eight of us in the app sick team and about 2,000 apps but like that number was some people say we have a thousand some people say we have 2,500 nobody really freaking knew we had a whole bunch of apps some thousands of that so we sat down and looked at like what do we have to do to do an assessment or understand the security posture of an application and so we started on the far side with the requests coming in or us understanding there's an app that needs to be assessed that's the intake portion right we didn't take those and then the triage idea what the triage position is I've got this work choice coming into our group I need to understand more or less how much love I need to give this app right if it's a low risk low use internal only and the risk profile of it is very low I don't want to give it a lot of my resources honestly I want to maybe just do a best effort on that guy but if it's a high profile high risk key app for the business you better believe it's going to get the full monty right so triage is where you kind of decide how much of the pipeline this application needs to go through testing phase is a way that we can all a cart handle testing this can be threat modeling dynamic static whatever manual what have you all that happens in the middle of the test position the key thing is that all of those tests need to output in a format that can be put into a vulnerability management database where you normalize and gather those together and you have one source of truth for the vulnerabilities and your app psych program and then from there since it's normalized data you can pull out reports you could push it into your GRC system you can push things into JIRA you can use this to filter out false positives before they get pushed into JIRA right and so having those things I've honestly stuck together but I want to find a better word but I'm just gonna say stuck together and the vulnerability management database allows you to do reporting and stuff in one sane way I don't have to report one way for my static one way for my dynamic one way for my manual Eldar test and reporting comes out the same and it's great it's super crazy fantastic for medics yeah as our primary source I really wanted them to be swappable and have an abstraction layer in front of it that was my entire goal because and I said this before but respectfully though I mean some of these tools are great and then they become less great about right and so I want to be able to talk about any of these tools and my envision was okay let's look at a repository plus we know the language languages that are used for particular Rico and then let's be able to run a suite of tools against those than a per curiam into one one centralized database we will be able to then normalize those and then spit them out and that was to be are pretty much a lot of our own even think so a lot of these tools to don't even have create the eyes right then what able to take those results and we're able actually run them and then push them into the same same spot because there's a lot of really awesome command-line tools Heather I want to be running those hoc so that was our goal is to get these into one place and games and questions are as a book I have lots of those right and we did we switched out our desk provider in the middle of this at-at Pearson and it was no big deal because there was an interface going in there was an interface going out and as long as we can write glue code to match those will buy you hello new one done that's a great point thank you for that so what do we want as an ala carte selection of choices but with we want ala carte when you come in in terms of how you're gonna assess but we won't need a reasonable number of choices right so it's sort of like you have the best analogy ever you go to Chipotle or what have you right you can get a burrito or a bowl but they only have so three things a mead and they have some cheese and they have sour cream and they have right so it is a burrito built your way but you minimize the number of choices and in the case of a knapsack pipeline you ideally tailor those choices to the risk profile of the app so the ones that are more risky get more loved and and the ones that are less risky get less lovin just because you have limited resources so what color is your pipeline as we sort of went through several iterations I'm going to talk about what we are sort of calling Gen one pipelines which is looking at this is straight out of I believe the DevOps handbook look at your team's purpose or at least inspired by look at your team's purpose the processes which aid that so for us oh I got I have to this is a mandatory quote for me this Debian quote like spending time optimizing anything other than the critical resource is an illusion and that's a really fundamental principle right if you've got a pipeline and you're fixing things down here but it's slow up here it doesn't matter how fast this one is you're never gonna fill the pipe before this when you got to fix the early one so you have to spend a little bit of time thinking about for your team and your resources what you can do what's critical in my opinion the critical resources is the app sect team like please raise your hand if you have too many people on your app sect team Oh shockingly once again no hands raised like I'm never gonna have a hand raised I've never had a team that was big enough so the idea when we first started doing this pipeline was there's eight of us and there's thousand 2000 developers how can we make make the work that each one of us has to do be that much more effective so we automated things that don't require human brain like setting up a desk scanner it really doesn't require a human brain right you need a URL you need creds you can usually if you have a decent API on the thing you can send those in being an API maybe you have a few pages you don't want it to hit that can also be sent into being an API like you don't need a human brain to do that stuff why don't we automate that right when you automate those non human brain things you've radically drive up consistency right because now the set up for let's say your DAST is the same for every - once you provide those baseline details right done now the really nice thing is we had crazy good tracking of status of things which I've never had before this is the first time in my 15 years of being an app sec but I actually knew kind of where everything was because we had these well-defined states it was really easy to track where things filmed and we had radical increase and we'll talk about at the end of flow through the system and the nice thing is it really increased our visibility and metrics because now we could say you know what I know that an app about this big because I've been in the industry for a while is going to take three days maybe but now we had numbers a - thing takes us long the SAS thing takes this long a manual thing takes us long and we have numbers behind that not just like from the hip industry experts experience talking but actual real numbers and we definitely reduce the friction because we started just pushing things directly into the bug tracker which was huge so here's this this looked rather like the last lad I showed you but this was our first iteration and really what we were focused on was making our team as efficient as possible it was very inward ly focused this really because there I've had a lot of questions when I've done these talks before well how did you connect to the CI CD or the other people and for our iteration one honestly I didn't care what the other people were doing I needed our house to be in order first and once that's an order and we're really moving fast then we can start taking on these external dependencies in essence by talking to other dev teams and bringing them on board and being quick about it because the last thing you want to do is convince someone to allow you to be in there see ICD pipeline and then make a mess of it right so get your house in order which we did with the sort of Gen 1 pipeline and then look outside yeah Asian security work so obviously it's like look I can find bugs all day but if I don't have a lot of my process and it don't make sense but I welcome my process why wait to bother with this so you know we know that so I actually spent a good deal [Music] so yeah and we've changed the way we presented this because of when we initially presented so we said this is how we did it and I had many many people saying so I have to go get this piece of software to do this I'm like no like please don't get the software that works for you like we at Pearson we had service desk and if that's run well that would have been a great thing to do intake that was not the case particularly at Pearson and we did a new Service Desk right I had it somebody in a training idea about this who said we have service desk and we have great admins and it's super easy to use and the whole company uses I'm like well what the hell use it like come on write this this is it's it's by this talk is very interesting to me and the kind of tricky because I don't want to be prescriptive it's much more of a this is a general idea and if you keep these concepts in mind as you make choices as you're laying out your pipeline you'll be in a much better place but you're right it's very it's very custom so once your house is in order right look outside your team's purpose and this is where you're reaching out to dev teams that are doing CI CD or that of some kind of automation because now your internal processes are saying you can actually reach out and integrate with others so the traditional idea would be stick something like gauntlet or dependency check or what-have-you right in somebody's CI CD and just write a little bit of glue code enough to ship it into your pipeline right now you have external peoples processes dropping into your pipeline and adding to your ability to see the state of the security of your group in Pearson I forget what outside some other group had a copy of a shoot lwas the dynamics a scanner from the people who died think of ah anyway somebody else had a dynamics a scanner that wasn't even in the security group and we said you know what you've got that that's great can you feed us results and we took their results and added them in there because it just gave us yet another view of what was going on and this the whole suite of things that we were in charge of looking over and so this ability to sort of be able to just plug in and if you do this right it's a rest request right and that's pretty dang easy so you finish a result you send a request in and done right you can also do things like have web hooks happen to cause actions when a one of skåne fires aah and then the other side of it is weaponizing jenkins right and this is a case where if you are weaponizing jenkins my advice to you would treat false positives like anaphylactic shock like treat it like if you're very very very allergic to bee stings a false positive is getting stung by like fifty bees right it will kill you and it will get you booted out of the pipeline really quick if you break the build for a false positive like you might as well just not gonna work for the next week or more because no one's gonna trust you and there's already not good trust usually between dev and security teams so actually treating giving them non-actionable results is really not helping you so make sure that whatever tests you do it is ridiculously accurate I would give up doing a test in a CI CD if I had any kind of idea that it was gonna have false positives with any hi at the mouth right it just doesn't make sense pretty good if you're in a build breaking CI CD case right things like SSL cipher checks well those are very determined I mean it's very determinate right you can know they have TLS to here's a suite they don't have these ones right those are tests you can put because the false positives really aren't there do they have the right security headers those are also simple tests you can make that are very very accurate and then think about them between health checks and scanner items and this is another one where the health check II kind of things and once again headers and kind of normal do they have the HTTP only on the cookie these are very sort of baseline security items those can easily be run all the time particularly if you're running your own Jenkins because that's another fun thing sometimes you don't have access to Jenkins you have to run your depending on political environments but the idea of changing the idea of taking those things that would normally be part of a large scanner particularly a dast scanner run and cherry picking out the ones that you can do all the time and they just run them every day every week depending on how long it takes and with your Cadence's run them regularly and just consider them as health checks should never be yes should never be green yep and then if you have a probably a more sophisticated group or AB or a program II kind of AB psych team right you can take specific issues right a very narrow defined test drop that into your CI CD and let Jenkins run it regularly and then you'll know when the thing gets fixed and so you can actually take in a finding of something like there's cross-site scripting in the login pages pickings to follow the wind write a simple test for that right that's a little bit of Python or Ruby or pick your language it's nothing you know whatever you want to do to make a request and see if I can cross that script it put that as CI CD run that and you will know the minute it gets fixed right you can get a ping in essence from your Jenkins box that says hey this cross-site scripting thing is no longer there and you can actually you had this where you told the dev team hey that last release fix this bug which when does that happen like never that was really cool and then cadence is another interesting problem you have to think about how long tests take and whether or not they're worth doing sort of a build breaking CI CD or outside of that process so for the longer running tests and we had this really bad at Rackspace we had lots of really nimble teams there were certain tests that just took too long to run and we couldn't make them quicker so we just ran them every week and you know what maybe they had ten or fifteen pushes to production in between that week that I'm scanning and that's ok because once I find it guess what they're gonna fix it really quick so yes maybe my I am not testing release but I'm testing every five or ten releases that's certainly better than testing every month or something like that right so there's this depending on how fast your teams are going and how fast the tool will run or your test will run you have to sort of pick these cadence issues and I wish I was good enough at math to give you this magical formula there isn't one just use your best judgment that's certainly a case where you have to sort of look at tests and it's really about taking the things that we would normally think of as the big scanner run and making the bits right break them up into individual tests and pull out as many of those as you can and run them individually and that'll save you time the other thing is if you're really clever all those things that you pull out as health techs turn them off in your dash scanners right if you're already checking us a cell every day via Jenkins turn that test off on your dash scanner why do you need that again well I took it twice right so you can actually make the desk and it's run faster by not running the check so you're already doing oh and here's what I'm trying to get to be able to do at Oh wasp I want to take project repose pull code out to Jenkins push those or launch doctors based on the code that I'm pulling use something like zap or other tools also on a doctor to test them take those results push those findings in to defect dojo which will be the source of truth and then from there I can do fun things like push notifications the slack if the project has a slack channel or push validated findings into JIRA or if they use github get have issues wherever they need them so this is something I'm trying to get done at a loss but I haven't made as much progress as I wanted to in all honesty but that's where I'm heading because I think that would be one I think it'd be really cool to we're using our own tools to test ourselves which i think is really hilarious I don't know if you remember a wasp arisin was a java static analyzer I pulled that down and talked to Paulo and I said hey if you ever said I found it bug with your with your horizon he said really what you do when I said well I downloaded a rise in and I compiled it and I ran it against itself sorry who you wrote a java tool to test Java I'm gonna run it against itself and just a quick word defect dojo had a I'm gonna move a little faster defect joke dojo to talk yesterday if you didn't go to it you should check it out it's a Noah's project it's fantastic there's a great single source of truth and it's gotten a lot of velocity and then evolving apps it faster oh well oh my goodness I'm still far ahead I need to go faster I've been rapping too long so what went hand-in-hand with railroads well that was a telegraph right Telegraph Wright was a way to speed up signaling now you had fast travel you also had fast communication and it used the common language Morse code right to transmit those messages alright let's look at automation right it enhances the benefit of an existing pipeline it follows the same path a ie it's consistent because it's following a bit of written code so there's no human interaction there and it follows a standard protocol rest in HTTP oh and please my little to cry out to everybody if you talk to a security vendor today or ever ask them to please have a sane rest interface for their tool I will no longer look at tools that don't have sane rest interfaces because I can't automate them it's a pain if you can't automate them they're not in my pipeline why do I have them so please poke the vendors ah here we go chat ops securing tools to check so again here two three so I enjoy trying to hook up tools to slack or chat or whatever because I think of it as a much easier way and just firing of those plus I want to be able to I get and I believe here that's a good a bad bit trying too much smarter really and so you know here's one thing she liked what I did this actually kicked off the scan music chat and then the web the web actually in the next channel I'm gonna call it comes back in and says hey look you're being attacked and that's always being done in your chat and you know I think that is really we're ready to go when I look at trying to engage my developers I keep saying making it personal astana plate when I have tried n development and actually knee above big security like you know isn't good right let's admit this right it means to do right and especially what you want to just put out some features and you're gonna be spending more time doing it correctly which is the right way to do it but I want to make it as easy the campus of it and even just an example here this way and then actually this is a true life story they have lots of cherry and I'm like okay let's do that something I like no that's Kurt's good what's cool like tenure and so then I look and see oh no that's not enable as well right and why aren't these tools being used is because they I was think they're they're false positive rates too high and then you guys have gone through that sort of thing and so you know looking at a better way to signal exactly what you're looking for then that told me come accept Pop's can see that they have I think the other operation so much easier yourself and I'm going through this right now where it's like I'm trying to solve a lot of additional security tools but to get a lot of operational yeah and this is this is somebody poking at the org and getting banned for 24 hours which happens all the time and I just get a little there's if you go to our public slacking go to the infrastructure infrastructure blocked IPS and watch them scroll by it's amazing how many people like fire up door buster at WWE don't work like come on really where I try different things see if it works and this was like super like everybody used it but I was expecting a different way rather than having a steel lagina nobody looked at it what could I put it in the chat can we have a chat ops where you could say hey look never get success there it is even if I'm the only one that used it and say just add technique the developers say over here here's another blackberry here's what a point to it or just static stuff cookies well the thing I liked about this is we did this at Pearson and we were all over the globe you know all around the world different developers everywhere and I'd like to sleep and so if my time 3:00 a.m. you want to ask me about cross-site scripting well you can ask the bots because it's not sleeping and at least get some rudimentary information until I'm back in the office which was nice if you know where it is github like why should you have to do all this stuff so that they're like I think further motivations a bit easier where you could just say hey what's Candace Revo and it's gonna create a project that's going to add to our source of truth and it's good [Music] we - installed yes sadly [Music] and push the results to s/3 s/3 would then we would adjust those results that way so like 10 there's a lots of different ways to do it and I have to say that at all yeah yeah we have to iterate all the time at half of the fact that we had our pipeline going well let us do these fun chat ops things because we already laid the wafer in our case our static tool to dump it into our our vulnerability management database so that was there so now it was just a matter of yeah made it easy yeah oh well we kind of settle this stuff already it does effectively scale you can obviously run tools anywhere in the deployment is silly easy with docker but you guys know that if you play with docker we're tight on time a bit so we go ahead take your security tests you know and you do it different ways and this is obviously that's you are security Stanford is distinctly the pipeline is code approach right so maybe greater security to have say hey look here's our security test this go here and now you've added security to your yeah and then connect your own pipeline experiment so this is sort of I call it if you want to get if you want to as I say come down out of the traditional offset trees using my evolutionary theme again right to do apps ik well I really think you have to write code I don't care what language it is I don't care how nice the code is I don't care if it's really crappy Q and I've written loads of really crappy code you just have to write code one so you understand the travails of developers if you've never been a developer you should at least bust your knuckles a few times that's been a day finding a semicolon and realizing there may be better things to do with your life but like we've all been there and if you haven't you should because then you can tell that were story when you're around a bunch of devs and they kind of go oh it's another brother who fought a day for a stupid character right but you really should do that and like I said I don't care what language just programmed something the other thing is most of the stuff we've talked about today with pipelines is glue code it doesn't have to be production ready multi-threaded mutexes yadda yadda none of that like simple simple glue code I'm going to pull in from this I'm going to munch the data I'm going to send it to that so don't worry about having to do these ultra leet sophisticated algorithm yada yada yada simple simple glue code most of the stuff we wrote is and under 200 lines yeah just simple little things to get you through what would help me get back into it was that I actually did Travis integration at the security test at a non functional test all those things you want in it and SDXC a pipeline and then maybe think wow so these are all the things that my developers are contending with and it's not just me talking and saying hey look they should do this I've done it and I feel I feel your pain but I also see that you know how do you make good suggestions for security yeah absolutely absolutely yes thank you I had to get over my embarrassment of I mean we're right some glue code and some of it's a little embarrassing but yeah commit to github you make one up in 81 but gave the check for me was like getting over that hurdle of beyond don't push it out there and then you know somebody might not use it with my smart ideas saying hey you know I can make that better for you the presence of technical will be seen especially we keep that go and do some things which has been fantastic yeah um quickly I'm gonna run through some case studies very fast cuz we're short on time and I want to make sure we get to the last bit absolutely pipeline company one they served Kirti findings turn him into tests much like I spoke about early about weaponizing Jenkins they added those tests like I said to Jenkins they ran them regularly they turned green when they were fixed they tied alerts and chat ops to those fixes and they had two FTEs that were able to assess 35 apps in one year so that's pretty good if you think about a normal sort of pen test window and the fact that they couldn't get regressions on existing issues because they wrote tests for them because you don't ever turn those off they just don't always green right so no big deal let them run and if they do actually ever go back to red you've had a regression you can find that stuff out the other one was company two and 2014 they did 44 assessments in 2015 they did over 200 and the reason the over 200 isn't known is because they didn't start counting till March so they're kind of a little bit of guest there but they did created a pipeline that launched in March they lost a couple of FTEs during that year but still made speed improvements oh yeah laughs well yeah somebody got half assigned I know we cut somebody in half it was it was a mess no anyway yeah sorry the following year they went from 200 to 414 which is a 2x improvement roughly speaking and they also lost even more people and that people weren't backfilled because everybody knows how easy it is to find AB sick people so overall if you look at it oh it cut off the slide it's a nine point four increase from 2014 to 2016 by doing this pipeline and it took about three months of metal work so to speak to get the pipeline in place but once it was in place so speed improvements just kept coming it was it was really surprising yep that's true that's a huge point back into the right or I don't code yet so I can look at false positives in the knowledge repository absolutely nice shark mindless changes change myself [Music] [Applause] it's not you and maybe what you can recover I say there's hope right so just think of that that's it