Managing your pipeline for security made simple with airSlate SignNow

How to create outlook signature

hello my name is chris reilly i'm one of the organizers of devsecops days rockies on october 29th 2020. you are about to watch one of the sessions from the amazing event i hope you enjoy it and please reach out if you have any input you want to speak in the future or just feedback about any of the sessions you've seen on youtube thank you and enjoy so how do we get started um so when you're looking at pipelines as a whole um we try to build um pipeline from scratch and and you know we all have likely read the phoenix project and um you know certainly it goes back a number of years but how many more steps are there to actually get to have pipelines so pipelines are a pretty broad thing we all know them as for deploying code our application codes certainly where they've started but where do i start what tools do i use um how do i want to accomplish or what do i want to accomplish in that pipeline and then clearly how secure do i make the pipeline sometimes we focus a lot on the applications and we forget what's actually in the pipeline because the pipeline can contain absolutely everything you're about to deploy and also connect all the things that you don't want anybody else to see um what level of permissions do i need to give to developers so if i want to run devops but i want to run it in such a way that the of the organization doesn't necessarily need um uh right access into develop into the production systems how do i do that how do i still give developers access but then not allow them to actually take down systems and then what permissions do i need to give daily devops engineers so we're going to go through a variety of different steps um starting with just the basics of devops principles which i'm not clearly not going to explain here and then move on to how do we actually create pipelines within them so um a couple years ago i saw this chart and i know it continues to get updated um which is this uh you know it's it shows the complexity of of picking devops tools and we kind of all know it and we pick our own and we choose a variety of things this periodic table sort of shows the various things from security to testing to continuous integration but the first thing is tool discovery and tool discovery is very important but once you've chosen your tools you need to go through those requirements of what you might need from a security perspective what do developers need and then what do you need from an operational perspective and i'm going to go through a case study where we built pipelines from an end-to-end perspective we secured those pipelines and what are what were the end results and in doing that um the security requirements were fairly high on this particular you know particular set of customers and they're done in such a way that ensured the least amount of touch on their secure data least amount of touch on their secure strings and the least amount of touch everywhere we possibly could and but ensure that we could have rapid deployment and wrap a rapid development so how do you do that how do you mix the don't touch the need to still do all the devops principles and to have automated pipelines and what tools might you use so i'm going to pick a few tools i'm not going to pick a lot because we don't necessarily need a lot so how do we deploy so i'm going to start in some of the basics of of building a pipeline now you would think um in the case of aws and that's we're going to focus on um how do you build a pipeline or why do you need to worry about your account structures well you should we should worry about your account structures you should worry about how your accounts are laid out and i'm going to go over what we would recommend in the account structure so once you've created those accounts or have them or need to create additional accounts then what pipeline patterns are going to span all the potential use cases for businesses and we want it to be touchless and what i mean by touchless is that i want to be able to deploy code into my repo and then i want it to propagate all the way through to the actual servers that or the databases whatever it's needed now the thing about it is that there are some things that you have to do so but in the secure world you want to make a touch list how do you take those things that you think you need to touch and create and make them touchless and how do you automate processes that might need a step in there that you really don't want the developers to do or the devops team to do so we'll go through some of those things um what are the precursors for creating those pipelines that can be no touch and they can be fully automated how do you automatically create or store or manage keys and secrets inserts because you want those automatically created as well and and that naturally there's some things that in case of aws they don't necessarily support the creation of those things through automation scripts release cloud formation and how to create the guardrails or what guardrails should be creating now i'm not going to go through all say cis guard rails or cis os hardening because all those things are important for securing an infrastructure i'm just going to simply focus on what is required for a pipeline and securing a pipeline and then we're going to lock them all down lock down the rules and policies and we're going to force automation and secure deployments and pipelines and although it seems um like you will not necessarily get the productivity that you want um if you see if you lock everything down you'd be surprised once you're once you provided um fully automated pipelines where everything goes through a code a code methodology and goes through infrastructures code or databases code or codes code you'll find that there is a simplicity to not having to worry about the infrastructure that's underneath it and what's happening so the first is accounts as i mentioned it's important you set up the accounts they're in such a way that provide you the structure that's going to give you the most security in in pipelines so a typical aws landings on a control tower deployment is going to give you a user account it's going to set up a security account it's going to set up a login account those are really important from audit and security perspective but what do you want to do from a cicd perspective so what we do is there's a higher degree of security that that where you add an additional account specifically for your pipelines and your pipeline management your code repos because you can then control your access to those you can also give developers a little bit more access to create those pipelines to create the things that are necessary to run those pipelines so we generally will create a a fourth account within our landing zone outside of what's actually required so outside of anything that's that you're actually running applications in so that ci cd account is specifically designed to run pipelines to pull out a code command whether or not it's creating containers or serverless environments its sole job is to deploy whole code and deploy code now what do you have to do within that well you're going to have to create cross account rules and that means that you're going to have to as you go through the process of creating those core accounts in the top part you're going to create um cross account roles and you're going to create things can be assumed and little it's not a complicated role to create but in this case each of the app each of the accounts on the bottom part that are going to be in receipt of the application code or the infrastructure code or whatever you might be deploying in your pipeline you just make sure that that rule can be assumed by the cicd account and now you can pass things from one account to the next account and allow you to control what's actually in the cicd account and what that looks like is you've got a core tenant you're going to have your ci cd account your security account your login account you use account and then every other application account is going to fall underneath and be and be using those assets those four other accounts so you create a very strong separation between security logging and ci cd the next piece is creating that creating that inter interaction with with your repo so the example that i'm going to use here is uh say github so you know we sometimes talk about least privilege um and sometimes we implemented what's necessary to get the job done and sometimes they are the same so in the case of github for instance if you're going to create a secure connection from your code commit sorry your code pipelines in aws through to github and you want github to notify aws what are those lease privileges well um in most cases um well in the case of github it's going to set a web hook over to aws what are those privileges well they're essentially the repo privileges you just need to assign though now it's tempting to assign everything um but you don't need to you simply assign the least privilege even on the repo side which is really sending access to uh web hooks over to aws and then getting pull a pull data from the specific repo you can do the same thing within code commit again you do a lease privilege and you do those for roles and policies if you're using code commit within aws again it's tempting to give to give things admin roles it's tempting to do all those other things but you've got to remember that a pipeline is capable of doing virtually anything in most cases when you're building a pipeline and you run through the creation of that pipeline that pipeline's going to to assume my role so as part of an aws um pipeline process and and cloud formation process it's going to assume a role and when it assumes that role and you always give it everything which sometimes makes sense you need to make sure that everything is locked down to those that need to run it and also lock down in such a way that if they can't make mistakes in such a way of destroying production systems or destroying testing systems so you have to apply a degree of least privilege levels and policies to pull the code from the right repos and deploy into the right cross accounts so let's take a look at a single pipeline that is a um that's running multi-steps so single step and multi-step so what does that mean well that means if i am building an infrastructure in this case i'm building some core infrastructure that is specifically for pipelines so yeah there's a pipeline that's built but what we do is we specifically create an infrastructure that is for those pipelines and that that infrastructure um creates common kms keys so those keys are used for encryption it creates im rules which are specifically for creating policies to control access in that pipeline it's creating secure s3 encryption buckets and using those keys ensuring that the artifacts that come out of those pipelines are controlled and they're not they are encrypted so what you don't want to do is create all your code and deploy it in an encrypted ebs volume and sit with all those artifacts sitting in a non-encrypted s3 bucket because it was quote-unquote part of the pipeline so from our perspective flexity sets up secure pipelines contain over 20 steps to secure the environment just for a pipeline just to make sure that that code is safe and that code is deployed in a safe way so what do you do if you are doing a multi-step so in this case this particular pipeline that we would build deploys infrastructure's code so the first step as you would see up here it pulls it from github the second part is it creates the infrastructure associated with that code so in this case let's say it's an ec2 and a load balancer and maybe a small database it creates that infrastructure the third part of that pipeline is now building the code so it's whether it's building a php system or building a windows application it's going to build those artifacts it's going to place those artifacts again back into s3 which as we've spoken about is encrypted as well as locked down and with respect to policies and then we deploy it and we deploy it to the ec2 or deploy it to the ecr the biggest thing within this is there's no code in the template so you structure your pipeline and a lot of people will structure it such that their code is kind of not code but there is uh you know powershell code or other code that might sit in a template well that template can be exposed and that template could be compromised so in these pipelines the the the end goals which are ec2s are targets they're not they're if you look at this template this launch template there's no code in it it contains security groups and it contains tagging contains other things but it doesn't contain code it doesn't contain references to code either so once it's been tagged then you use features within the code deploy system to securely send code its way and we use ssm to leverage that code deployment so now i have a pipeline that builds infrastructure builds code deploys it to the ec2 but that cool core code that is doing that pipeline and doing that deployment actually doesn't contain any um any code in itself that is targeted towards those ects the third is databases code so if you're going to go the whole way and look at a look at deploying code well we also look at deploying databases code again as before we deploy the pull from source in this case it's whatever the source might be we deploy the infrastructure in this case it could be a database it could be two databases it could be whatever is required to build that rds ec2 whatever it might be then the third is that the database objects are built and deployed so in this case it happened this happens to be something that's running for a windows application sql server um you can also use other products that target postgres and other databases but going that next step which is we will build pipelines that allow you to do infrastructure as code as well as databases code now how do you secure that well just like before there is no manual connection to the database you will not see anything in here that associated with connection strings you will not see database code sitting in powershell and you also will not see any connection strings in the actual um github repository those connection strings will sit in a parameter store or sequence store and get pulled out and dynamically change so the code over here just an example allow you to substitute things that come from parameter store and put them into your code so if you were to look at your code on github you don't have secrets in that if you look at your code in in the powershell build spec or in your app specs there's no secrets there's no strings they all exist and they're only ephemeral in the sense that once that build occurs then you're going to be able to deploy them they directly go into the database and there's really no persistence of that there's no logs of it they're treated as no echo um and you deploy your database code in a secure manner as we as we look through it how you prepare or prepare your pipelines well as i said before you create kms keys so you make sure your buckets are encrypted you make sure buckets are transient you make sure you have common artifacts um very commonly artifact buckets are not encrypted so you encrypt your abs or you equip your s3 but you don't encrypt your artifact buckets and their artifact buckets are controlled by rules and policies how do you do keys keys are really hard to do and they generally are done manually but what we deploy is we deploy systems allow you to create rsa ssh keys using the custom resource that is now available within um cloud formation scripts it uses a combination of cloud formation lambda functions security groups and serverless systems to generate the keys and deploy them back into parameter store and then they're referenceable within cloud formation so in this particular cases code at the bottom makes a call to a particular secret of secret lambda secret creation uh lambda create a key pair the key pair comes back um which can be used in the cloud formation script and then it's permanently stored within parameter store so you not only have used it in your cloud formation script you place it in place that can be retrieved and then you can use it on an ongoing basis whether or not you're using it for eks or yes windows ec2 or or linux east jews whatever it might be and your pipeline now does not have a manual step of creating creating keys how do you do that for connections and user strengths well again a similar thing you create a custom resource associated with the encryption of your of your strings so the encryption of your strings um uh is done from a person that might be sitting over here there's a cli command that they run that creates a it generates an entry into cloud formation that stores it into lambda function or calls the lambda function that encrypts it and stores in the parameter store and then it's referenced within the cloudformation script when you actually need that that strength and what does this mean there is no non-encrypted string within any of this it's only as understood and known by the user and there is no way to for someone else to go in and say i would like to go see because remember you can lock uh you can lock your parameter store and your sql store down to a very small set of users and now those secrets are only known by the person that entered them and they don't even have access and write access into the system how do we do parameter substitution again spoke about it a little briefly secret stores are stored in primary store or secrets manager and we use a touchless system and we rerun and constantly uh improve the system to ensure that we are substituting strings from uh from secure locations and not embedding them in code and not embedding them as cloud formation scripts or any scripts associated with that um we've got a few minute few minutes left what kind of guardrails do you put up as i mentioned before there's a lot of guardrails that you might do in respect to cis standards but what do you what are the guard rails that you want to put up to secure your pipeline so in this case you want to make sure you've got key rotation sure that your users that should be available to pipelines you've got rotation on that make sure you've got s the proper s3 bucket controls you got to make sure you have the proper encryption controls and policy checks now what i show on the right hand side it's really easy to set those up in this particular account i have over 100 violations in those particular errors and some additional ones well yes you can run config but you got to link it to remediation so most people don't then link it to a remediation and if the remediation is only alerting an email that's phenomenal because at least you've got the notifications you aren't coming back to the console and there's lots of other ways to remediate but we certainly recommend you remediate uh at a minimum to alert on email so in the end what do you have you have a system that spans across those four core accounts the specialized cacd account you've got your your dev your staging your production accounts you've wrapped that all up into secure mechanisms and pipelines and mechanisms to use and create key securely and use them securely and not have code running everywhere and you create pipelines that really if you're to look at the pipeline code you would not actually know what's being built and what is within it you might know an ec2 is being built but you won't know what code is being put on it what strings are being used or any other types of data that's being shown and what do we