Empower Your Business with airSlate SignNow's Deal Pipelines for Organizations

How to create outlook signature

[Music] hello everybody my name is Roy oshirov uh I wrote a couple of books on on unit testing and a book on leadership and I'm currently writing a book on Pipeline driven organizations well I used to I guess trying to do something to me okay um everything you're going to hear now today is based on true horrible stories from organizations that I was Consulting with or that I was working for um so it's all based on true sadness okay if you want to hear more or read more you can go to pipeline driven.org or fivewise.com or just follow me on Twitter and Tick Tock no not Tick Tock okay so our agenda today we're going to talk about the difference between waterfall pipelines and Cooperative pipelines Co-operative pipelines are this thing that I'm calling and that's kind of the name that I found that I call it co-ops I think it's the evolution of where we're going in the world of continuous delivery uh we'll talk about how Netflix kind of moved into that um pattern without calling it co-ops we'll talk about what skills we would need to actually succeed in achieving true continuous delivery so we'll talk about the idea of full cycle developers and probably q a we will not have time so the first thing we'll talk about is cooperative pipelines and how they actually enable true continuous delivery and I'll differentiate true continuous delivery from continuous delivery or automated builds because if I asked if I asked everybody here do you have automated pipelines today in your organization a lot does anybody have more than two more than five more than 10 et cetera et cetera not you yourself your team but in the organization sometimes in large organizations you can have hundreds of them for different purposes but are they actually improving they're actually helping us so the next question I would usually ask is do you if you have automated pipelines are you doing continuous delivery today raise your hand if you are okay that's a much smaller percentage than the people who raised their hands earlier um I'll talk about changing the rules because I think that's where the root cause of our pain is and why it's actually good for us to go through the pain of transforming how the organization works okay true continuous delivery to understand true continuous delivery or how to move there I want to introduce you to a concept and maybe some of you have known it but I think it's relatively unknown in the software industry theory of constraints has anybody here ever heard that term okay quite a bit that's good I got introduced to the theory of constraints from the book The Goal which I think a lot of people who know the at least the concept have read that book uh by eliao goldrat and the theory of constraints that the idea of identifying limiting constraints and continuously reducing them or optimizing them until they are no longer constraints what is a constraint or a bottleneck has anyone here ever had a bottleneck well most teams today in organizations have bottlenecks and they could be related to knowledge for example only one person knows how to do something we call that well I usually called it a bus Factor a bus factor is how many people have to get hit by a bus for the team to stop working bus factor of one is really really bad but a lot of teams have a lot of bus factors because of knowledge but there's also permissions for example only some people are allowed to enable something to go through different pipelines or to enable delivery or to sign off on things so the theory of constraints is kind of a tool that I've adopted in the past few years that has really helped me kind of gel my thought process on continuous delivery and transformation and no longer think about it in terms of agility or all that stuff but I really think about almost everything that I do today in terms of bottlenecks it's really shaped my uh my thought process on that so the book I'm going to use that I would recommend to read or at least the audiobook version is beyond the goal that book really changed how I think about it how I think about these things and specifically he was adapting theory of constraints for a lot of different Industries and specifically for the software industry he introduced the concept of critical chain so I recommend that book as well as added bonus reading okay so the theory of constraints one of the powerful tools that we can use is to ask these four questions and he was talking about the idea that when you introduce a new technology into an organization you can't just introduce it and expect optimization you have to ask these four questions what is the power of the technology in other words what does it enable us that we couldn't do before second is what limitation or restriction or constraint or bottleneck does it actually decrease for us we'll talk about it but let's Con in the context of a car can you imagine what constraints we had before we had cars and then cars were introduced imagine there were no motorcycles okay there was just bicycle everywhere okay so the power of the car is that you can drive very very long distances right what is the restriction or limitation that a car can reduce for us is the length that we can travel what rules or processes before we had the technology did we have in place because of those limitations so if I only had a bicycle one of the rules I probably would have is only find work that is close to my house right because I'm not going to travel for different cities on a bicycle unless I'm Norwegian of course and finally what rules should we use now instead with the new technology and this is a very important difference a lot of organizations they introduce new technology for example pipelines but they don't change the rules they keep the old rules based on the old limitations and so we get a lot of these things what limitations did we used to have in the world without pipelines without automated builds well here are some examples of limitations in a world where everything was manual it took a long time to run tests right A lot of times the tests themselves were manual uh you only had a specific amount of environments available because you were very very low on resources it cost a lot of time and money and effort to raise those environments I'm sure that doesn't happen today in many organizations right today everyone has hundreds of environments right never mind uh how long does it take to set up an environment was a huge limitation uh who knows how to run the test who knows how to load the environments so there were specific people with knowledge in the organization how do we know tests have failed that was a huge limitation only the people who ran the test knew that they failed and how often did we know when did we get the feedback the feedback Cycles were very very long um how much information do we have on failures we don't have log files we just have maybe screenshots or somebody has to write all that stuff so we have a limited amount of information and deployment time can be very very long rollback time can be very very long these are examples of limitations in the world before Pipelines if we combine all that stuff we can consider it one huge limitation we do not know at any given moment if we can release the software in a world that is before Pipelines there was no way to know on demand if we can release the software today so we had to do everything in our power to try to overcome this limitation or to live based on that limitation so we created rules rules that made sense at the time for example look since we don't have all that automation then you'll only run the test because there are a certain amount of people that can run them only specific week or a month because we have a limited amount of people we have a limitation um only run the test on one of the branches we don't have enough people we don't have enough resources we don't have enough time to run all the tests in Peril there's no way that can happen we have to use code freezes and create extra branches because of all those limitations because we don't know what is the status of the software at any point uh we have to get QA to approve the releases because they're the only ones with information on the status of the test they're the ones only ones that actually know kind of what the hell is going on so of course we're going to give them the priority of making a decision because they have the most information in the organization and of course because they have all this important work we definitely have to separate them and not let them do anything else except continuously hopefully run our tests and give us that feedback they were so so limited and the job is so difficult and takes such a long time we absolutely need to separate them so we only release when they say that it's okay and our definition of done because we have this limitation will usually be okay looks like it's working um but let's not repeat it for example in many organizations one of the rules would be only test the new stuff we don't have time to test all the old stuff we assume that the old stuff is okay and worst case will create a bug item later on and because of all these pieces of knowledge that had to work on different pieces and the lack of knowledge on the entire system we put different people as gates in the delivery process because we had no information developers has had specific pieces of information QA had specific pieces of information security compliance Ops well there were there was some kind of Ops but those days they might have been called integrators Etc um because it takes such a long time so we have to batch a lot of stuff together to deliver it together because we cannot spend that type of effort for very very small things so let's create a huge bucket of things and then test the whole stuff because it costs so much time money and effort and of course everyone does exactly what they know how to do nobody does nothing that they don't know how to do because we don't have time for that we're basically in survival mode in this world because everything takes such a long time and it's very risky and if you have a problem we're already in release just create a bug item we'll take care of it later on so those are examples of rules of course in your organization you might have more rules different rules but those rules were built in a world without pipeline without automation without a lot of knowledge but today we do have pipelines we do have automated builds we do have all that stuff does that actually help and that's our core discussion so here's as a consultant when I go to organizations here's what I see a lot of times I ask so do you do continuous delivery oh yeah yeah we're you you're not going to believe how many pipelines we have crazy like everything is automated that's cool let's look at the pipelines well developers have pipelines and that pipeline does run on every commit where does the pipeline end usually in a developer environment I'm talking about larger organizations or even mid um but how does it get to QA a lot of times QA will even have their own little pipeline but developers decide in many organizations when to merge either between the branches between Dev and test or to move to different environments developers have to sign off on that QA will have their own automated pipelines some of the tests are automated some of them will not be automated but they have their own little pipelines with their own dashboards and that's cool but how do they get to delivery well Ops people usually take over they have their own automated pipelines and of course before they even can deploy to specific environments like production sometimes staging sometimes even earlier um the QA gets to decide the test leads get to decide whether a version is ready not the pipeline but a person looks at the pipeline and says I think it's good a lot of times we need a person to look at the pipeline because the pipelines usually have very large integration tests and they create a lot of noise which means that the pipeline is usually red but the people who know the pipeline really well I call them the build Whisperers okay they know the pipeline you can bring one of them and say look is that build really red and they will go and they will really listen closely to the build say uh-huh yeah no that build is green it just looks red but it's actually good you're good to go okay so the build whispers are vetoing the build they make the decision the build is basically just noisy at some point you just look at the screen it's always red and you just look to the test to the people who run that build whatever adopts Ops also get to veto what about security security also gets the video and each one of them in many organizations have their own Pipelines but what is between all those pipelines there's people and those policies that's what I'm talking about that actually prevent us from actually achieving through continuous delivery because the pipelines don't actually change our policy they do not change our process they're just automating things but we haven't changed the rules what do Piper what do pipelines actually offer us because with a pipeline technically you can do endless repeatable tests you can have endless environments you just throw a bit more money on it you can do multitasking at scale and test multiple branches in parallel you can do full feedback and history for every test that ever happened ever in your code for any branch Okay so does that mean that we still need these rules when we have pipelines all these are the rules from before right um because usually we have pipelines but also all these rules and that's where I think we're going wrong we are still using the old rules and the old policies but we introduced a new technology so why does it work for Netflix or Google or Facebook why are they able to achieve it but many other organizations are struggling because if we take this pipeline right that I just described and let's flatten it out let's just look at it as a organizational pipeline right code goes from one side to the other with a lot of different department a lot of different videos in the middle the difference between that and what a Netflix would do or a Facebook would do kind of looks like this okay do you see it okay that's the difference hold on let me just make sure you got it that's the difference that's the difference it's always people I'm always telling everybody that when I do the training software is never the problem people are always the problem every software problem is a people problem and here yet another people problem if we could just get rid of people in software everything would be so much easier especially in Pipelines okay in fact we could even paralyze that stuff and run it you know automatically and then in that pipeline instead of people watching the gates and approving things we let people create or add tests into the pipeline so their job will actually change instead of saying this is good this is not good their job becomes I will teach the pipeline to make a decision if this is good or not good basically we are integrating our manual processes as automated tests and if those tests are passing then the pipeline should be green we should not need humans in that equation and every test can break the pipeline right you want security to decide if the pipeline is good enough add an automated test you want Ops to decide if the infrastructure is good enough add an automated infrastructure test you want developers you want testers you want compliance people you can even automate compliance checks compliance is usually very very simple make sure that some documents are signed somewhere or that there exist in a specific directory just make sure you follow the rules those things can be automated so this is in what I call a Cooperative pipeline a co-ops pipeline here's how Netflix went through this now I didn't work at Netflix I didn't consult with Netflix but I follow them so everything you see here is from Netflix blog posts which they published in 2013 Netflix was where a lot of companies are today Branch per environment manual merge deploy triggers right all these things this should look very familiar to a lot of people oh yeah we have environments we have like we have to move between them and all that stuff the other people in the middle of the pipeline um in 2015 a blog post from Netflix discussed the idea of dynamic pipelines where basically they've automated all of the Automation and verification tests up until the moment of deployment to production everything else became automated so no environment movements between different environments had to be done by people no manual merges the only time there was a this is from their own internal tool the only time when they had a manual judgment was in the production go no go manual judgment verify Cannery scores are correct and you haven't received any emails or something okay this is official Netflix documentation um so they were on the way but production was still uh far away from that in 2018 Netflix uh released a blog post about a new tool that they created that they're using internally called Kayenta and Kayenta does automated metrics analysis in production or in any environment that you that you choose it even has an API it was built as part of spinocker which is was another project but you can technically use it as standalone and what they wanted to do is to remove the human equation that from the person that looks for emails and look in the and looks in the analytics after deployment to production to say yeah it looks good let's deploy the whole thing so Kayenta looks at the whole thing looks with averages uses it uses uses statistic analysis on the metrics that you provide using Prometheus or a lot of different tools and basically makes a judgment it makes a judgment and actually deploys the entire thing no humans in the equation this is a no human pipeline so Netflix went from 2013 to 2018 which they wrote about it and you can assume that if they wrote about it it was probably used a bit earlier but they took them a few years as well to even you know get to that stage as well this was 2018 this is today's 2022. a lot of organizations are still in the first step where Netflix exists today okay so this was their uh these were the first three blog posts then another blog post in 2018 that Netflix released was the idea of a full cycle developer and a full cycle developer their job is to take part of in any of these processes design development tests deployment operation and support right developers are not just developers they're also testers there's also Ops they're also support developers or full cycle Developers because in a system where a pipeline makes all the Judgment calls everybody has to be related to the pipeline which means that everybody get needs to write tests that will run in the pipeline everybody needs to understand how pipelines operate everybody needs to understand how tests are formed and how how to write code because nothing is manual in that delivery process this is kind of scary a lot of organizations are not in that position today I call it sdlc 2.0 um and I I and I came across this after thinking about the idea of pipeline driven organizations and I was looking for examples of organizations that implemented because I knew that they were implementing A variation of it and I was very happy to find all these kind of public blog posts that talk about it so instead of saying look we want to automate something so let's call it devops we want to automate the testing so let's call it Dev test Ops we want to automate machine learning AI Ops we want to automate uh what else did I miss there's probably a I don't know there's a bunch more right so at some point you're going to get like this test Dev AI blah blah blah blah blah Ops like okay what's your role well I'm I'm that the Cyclops or whatever so I just call it a pipeline driven organization because ultimately there's going to be another thing tomorrow that you want to join into the pipeline so instead of just adding another tulip two or three letters into that name we're a pipeline driven organization that's really what we're trying to do and there are many benefits to doing this if you trust your pipeline and you have a pipeline that kind of looked like this or in a variation of this they're one of the biggest benefits is that when you move from the left to the right right so in a world in the before a Dev would approve something to merge between branches but in the world after the pipeline makes a decision whether the merge was successful right in the left QA would approve a feature or release as deploy ready on the right the pipeline decides I wrote here the words pipeline delegate pipeline delegate that's basically like a un delegate it's someone that votes on your behalf if you want the pipeline to make a decision about something that you care about add a test the test is your delegate all bills break when a test fails assuming that your organization has breaking bills some organizations have bills that are not green and not red they're like yellow or something don't do that okay the whole point of the build is to provide feedback that's actionable yellow is not actionable yellow is like I think it works and what do you need you need to build Whisperers again what are build Whispers they're the bottlenecks they're part of the bottleneck I want to add one extra thing right there's a bunch of stuff that happens here but has anyone here ever worked in the in in the role of a test lead or in a test department and they had to approve a release even not a tester has anyone here ever had to approve a release sign off on a release what a joyful occasion that is right sign off on this release that maybe hundreds of people have worked on for two months well of course I know everything that happens in this code I will sign off well actually the last release really had a lot of problems so sign off that this time there are no problems well of course let me sign in blood I literally saw organizations that create a paper and you actually have to sign it for compliance and how much stress is there to sign that piece of paper that well I think this is one of the reasons people burn out how can you even ask a person to be accountable for so much code that they've never seen with so little information we are not supposed to do that in a pipeline driven organization in a pipeline driven organization we have tests if you want to trust the pipeline create the test that you want to run and if the pipeline if the tests are passed we're already in production if the pipeline is red you don't have to approve or disapprove I would say that sometimes disapproving the release is even more stressful we are all we all need this release please sign off on this release so we can release it well it's not ready come on dude we have to release the software there's so much pressure it's scary what you you want to put a person in that position with all that pressure management is like swooping upon them like eagles find a release sign to release I would quit at some point fine we take the money first then I would quit so the biggest thing we can possibly accomplish at the beginning just by implementing this is reducing the amount of stress the pipeline is red the pipeline is red there's nothing I can do about it the test failed of course the worst thing you can do is to what comment out the test check off the test of course nobody here does that right nobody here has ever disabled a piece of the build in a pipeline driven organization you do not disable pieces of the build because the whole point is that if the build is green you deliver but the build can be read for a while but when it's green ah you should see when it's green it's beautiful okay so the only question left is would you let your build deploy to production and most organizations if I ask them they will say I don't even let my people that I trust the most deploy to production you think I'm going to let the build deploy to production we have Ops people their whole job is to say no their job they're defined they're they're measured by how much they say no they're measured by uptime they're measured by how often the software does not change Okay so that is an example of a policy change remember I talked about changing the rules we've introduced this beautiful technology very powerful but we're keeping the old rules if we actually want to invest in that we have to change the rules and this is what the netflixes are doing what did they do they changed the policies they changed the rules and that enabled them to run much faster but it had to create a different structure in the organization because there was a lot of skill challenge remember the full cycle developer we talked about so here's an example when I usually come to organizations and we talk about these Transformations and that connects to the previous talk that we had as well a lot of organizations are fighting with the idea of a pipeline driven organization whether they want to admit it or not they're on that road developers have already felt this if you're a developer and I'm assuming most of you are Developers you've already been asked to do a lot of stuff that usually you would not have done for example being a full stack going across different layers understand a bit more about how the pipelines work maybe write automated tests I really hope so right maybe start thinking about security a lot of organizations are integrating security it's really good but it's really really tough you have to learn a lot of things and also coaching where is the coaching coming in well developers are not the only ones that have to contribute to a pipeline right Ops also have to contribute Ops also go through this transformation how how do I know because they you they weren't called ops before so I know they were going through that transformation a lot of them just change the name but they don't change the rules so everything stays exactly the same they're just called ops but their job is still to say no but in this world in a pipeline driven World they have to learn a lot of things and some of them have already started infrastructure is code writing automated tests for infrastructure so learning how to code a lot of them do are not Developers automated testing is really difficult if you've never been a developer but you want to write infrastructure tests integrating pipelines a lot of them used to do Integrations manually and deployment manually now we're asking them to automate everything and of course coaching who would they coach they would go to they would coach the Developers right the developers need to learn how the pipelines work they usually never needed to do it now they do what about security oh security is slowly today beginning slowly in the past couple of years beginning to go through that process but they're still very much kind of in the old policies if you're an organization that has security departments today you know that they're they represent a very powerful bottleneck in the organization but in a pipeline deriven organization you want Security in the pipeline you do not want another handoff as much as possible you want to reduce it so Security will also need to learn how to play inside the pipeline a lot of times Security will have their own pipelines they know about automation they know how that stuff works but they're not integrating it into the delivery process a lot of times they do not do a lot of coding they do not write automated tests but they will automate using some tools and that's fine but they will need to learn how to integrate the pi with the pipeline and they would need to do coaching who would they coach well they need to coach the Ops people about environment security and they need to coach the developers about security and building Security in and of course I know that if I asked all of you will tell me that security have always been the most helpful the most coaching people in your organization today right they're easy to find you they're right there as you need them you need something security is there to tell you yes is that correct am I am I missing something must be in a parallel universe um and finally testers testers have been feeling the pain for the past five probably even 10 years the pain of automation the pain of learning to code but testers have been in that road and we've we had a gut feeling that they should be going through that road but now we can put it in a name or we're not doing devops we're in a pipeline driven organization so we want our testers to also teach the pipeline how to how to make good decisions so things that they usually test manually they will find a way to automate some things yes will never be automatable because they need human decision making but a lot of things are a lot of things are and there are different Notions of that you can have a delivery pipeline that is fully automated and you can have a discovery pipeline where you can add extra information for example Performance monitoring ux whatever but it does not hurt delivery even if it fails and you can have a delivery pipeline that has to be green just that notion of having these two separate pipelines can be very powerful because you have to find a place for the test that you either cannot automate or that are very noisy but either way testers they have to learn about automated testing coding test infrastructures because the organization is going to use them integrating them into the pipeline because they have to contribute to the pipeline and finally coaching who would they coach they would coach everybody because in a pipeline driven organization everything is a test everything is a test what is a pipeline it's one big test it's either red or green hopefully it's either red or green not yellow please it's a big test and it's made up of a smaller tests and if one of those smaller test is red the whole test becomes red so in a pipeline driven organization whoever knows how to write good tests is king and queen they are the people who have the most experience I'm not talking about people with less experience but people test this with good experience have been reading books about testing for the past 5 10 15 20 years they've looked at different scenarios that as a developer I might never even even thought about they look at the system from a different perspective than developers they can teach developers so much but not only Developers right they can teach everybody so this is a very small diagram of the dependencies between the different pieces of knowledge and I've put a color for each one because what you can see is that everybody is kind of starting to integrate with everybody in a pipeline driven organization and this is where transformation begins to be very very difficult now if we accomplish this there are a lot of benefits first of all just from a personal perspective we talked about less stress from a point of view of job security in a pipeline driven organization everything is test but also everything has code security is integrated everywhere and Ops is part of the whole thing which means that as you as you become more and more professional and you do and you become a cross stack developer Ops whatever it is you become more valuable to the organization either here or the next one and what I'm trying to say is that the next time that you're going to look for a job there's going to be people that have those skills and there's going to be people that do not and the people the people that have those skills are gonna be prioritized because we're moving into that world whether we like it or not I think we're moving into that world for the organization it can be a huge benefit because we're reducing bottlenecks we can create faster lead times faster feedback less bugs in production less bus factors and bottlenecks and basically achieve true continuous delivery like the CEO always told us so some examples of rules and processes for the new world pipeline decides the pipeline is the judge if you want juries create tests the juries will testify for the judge and if one of the juries says actually I'm not crazy about that stuff then the judge says well no we all have to agree um second we need to trust the pipeline to make that happen and if we don't trust the pipeline then our job becomes teach the pipeline enough so that you can trust it second thing third thing we run the pipeline as often as we can do we can allow it or as many pipelines as we can a simple example if your pipeline today takes three hours to run should you run it once a day or once a week no you should run it every three hours because the feedback Cycles are much much faster and every time you optimize the pipeline your feedback cycle becomes faster and faster stop thinking in terms of days and start thinking in terms of feedback loops that are continuous would you want to know that your code Works only tomorrow or do you want to know that it worked in three hours or in one hour or in 15 minutes even if you didn't do any other optimization even if your pipeline sucks in terms of speed you can still usually run it much much sooner than you realize or but who decides when to run it that's a policy that's a rule we can change it run the test on as many environments as possible instead of deciding that you run the pipeline for a specific environment run it those in as many environment as possible that's what this enables us we should use this but usually organizations have policies to only run things in specific environments and we think that we're just stuck in that type of reality but the truth is is that we're there just because of a bunch of decisions earlier that can be changed we're all learning how to develop software this is you can treat it as an experiment of pipeline driven organizations and I highly recommend using that word let's experiment with the pipeline driven decision making um any manual test is potentially entered into a pipeline as automated so whatever you can you automate of course there could be huge technical debt but as a vision as an ongoing effort this can be very powerful every test that you automate is a test that you stop doing manually which means that you actually save a lot of time the more you have to repeat the test the more time that you save economies of scale so you actually create a very positive feedback loop whenever you automate even a single manual test because now you actually have free time that you would never have had because your free time what do you do with that free time you automate another test now you have even more free time oh we can automate another one with that extra free time etc etc it's a very positive feedback loop but to do that we need to change the policies and the rules in many organizations if the pipeline is green in all stages we are already in production right some organizations do not go to production where you have different customers with different whatever but as close to production as possible um everyone writes tests everyone is a tester in a pipeline driven organization working small batches instead of working in huge batches reduce the amount of batches get the pipeline to run on very small increments if the pipeline is green that small increment is delivered of course that means you will have to learn how to use stuff like feature toggles Etc because you don't need code freezes anymore because you get continuous feedback but feature toggles become your friends and that is a very important skill but also feature toggles require policy change the first time I had to coach a team in switching from multiple branches to trunk-based development with feature toggles they were so scared it takes a leap of faith and he takes the manager that says okay I'm willing to experiment it is a people problem it is not a technical problem spend 50 of your time teaching the pipeline to make a good decision think of yourself as a gardener you spend most of your time teaching the pipeline to make good decisions and last two you spend time coaching what the hell that's uh okay okay spend time coaching others about your expertise that's communication is there a fire what the hell is the pipeline broken okay last one I promise come on we can do this don't disable a part of the pipeline to pass the build I'm talking to you and I'm talking to you and finally [Music] a red pipeline means no task the task is not done no need to create a bug item you keep fixing it until it's green and you develop it and then you deploy it and with that I want to leave you with one file well I'm just going to jump in because we're out of time okay finally Cooperative pipelines are the thing that I think enables through continuous delivery we have to adopt new rules instead of the old rules and it's in our best interest to learn those patterns if you want to learn more you can go to pipelinedriven.org my site or my Twitter thank you very much everybody May the force be with you [Applause]