How to industry sign banking oklahoma presentation safe
good morning everyone or good afternoon whatever the case may be this is Alan Chimel of DevOps calm and welcome to another webinar we are going to be getting started in about 36 seconds to a minute as we allow any late arriving guests to a chance to join in so before we get started you know many of you may have attended our webinars in the past but if you haven't we use the go to webinar platform and it's pretty full featured it gives you the ability to ask questions within the questions section on the right you know hopefully your GoToWebinar interfaces in the right hand side of your screen and you can ask questions in there and we have set aside time for questions we'll talk more about it in a second there's also a chat area where you can ask questions or panelist or the entire audience or comment we again we are holding questions till the end but you can write your questions in at any time and they will will stay there until then we'll as I said get started here actually it's about 1101 on eastern time so we're going to get going just give it another five or ten seconds as I see people are still still joining it I know hopefully what's a beautiful wednesday wherever you are okay all right let's get started hi this is Alan shelf editor-in-chief of DevOps calm and welcome to another DevOps webinar very happy to have our webinar today sponsored by our good friends at sevilla labs and today's webinar is continuous delivery and DevOps at rabobank very lucky to be joined by a great panel today and if we can go to the panel slide first from a rabobank manager of Linux UNIX infrastructure services we have sander edameh Sandra welcome to DevOps com webinars thank you and joining sander is a distinguished author at web at DevOps calm as well as the VP of products for zippy labs great fellow always interesting to hear what he has to say Andrew Phillips Andrew welcome pleasure mia XL okay before we get started I just want to do a little bit of housekeeping and if we can bring up the housekeeping slide yeah and as I mentioned to those of you who may have been here earlier or who came on a little earlier i would go to webinar interface does have a section where you can ask your questions you do we have set aside time at the end of the webinar for questions and we hope to try to get to as many of your questions as possible if we don't get to your question your particular question we do give it to our panelists and ask them to give us a written response which we will post to our page for this webinar on devops calm but my advice to you is don't wait till the end to actually type your question in because we do them as they come in and sort of chronological order so ask your question early and it has a better chance of being answered additionally this webinar is being recorded and it will be available for later playback via YouTube mp4 along with the slides it usually takes us 24 to maybe 48 hours to get that up on the site links to the slides and there were recording as I said well be out in 24 to 48 hours there is a chat section as well but we do recommend that you use the question section for your questions with that being said let me not waste any more time and I'm going to turn things right over to Andrew and Andrew if you want to take it from here yes thank you very much Alan and thanks for everyone who's taking the time to join us today i am i'm really looking forward to this presentation partly because i get to listen to to someone who is doing this for real in a very complex environment which is incredibly useful interesting learning experience for us we have Sandra I'm not going to steal too much of just under here and I think the main thing that that is so relevant to everybody who is attending about this particular story is that often the words continuous delivery in dev ops now the stories you read about the blog posts you read about and the books and so on many of them deal with companies that are plenty away in the kind of industrial landscape where you say you know they've got maybe a tech company that's a retail company basically subsisting on the website or whatever no the vast majority of IT happens in existing organization to the totally different setup and then have to deal with things like not just business pressure with regulatory requirements and and how do I go from where I am today with a mix of technologies to this faster more accelerated future that I keep reading about and and I think that the most important thing we can contribute in this particular education session if you like is to show people how companies that are coming from this background are actually doing this and that's why I think what's under is going to talk about is so relevant it's so important I think mean the other thing is that you know a lot one big question specifically that always comes up it's like yeah great if you're in a future star the website and then just great a couple of pages don't shop no big deal we are in an industry where every crashing let's remember the New York Stock Exchange just in recent memory and bad stuff happens when things go down we have all these other requirements we have all the security requirements we have complicated long long running change in control processes of trying to make this all stay up surely there's no way for us to adopt these fully automated pipelines that we keep reading about to me but was very interesting was to hear time they're talking a bit about how by adopting some of these processes they've actually managed to improve their capability to meet their other requirements that compliance regulations but as I said enough about me and let's hand it over the summer really looking forward to what he has to say thank you Andrew yes well for starters of course i'm working at rabobank as a manager of landis unix and websphere we are also hosting several appliances and the main focus of this story is about the websphere Java environment so and the central theme about the story is the triangle between cost risk and time because that's where everybody is in a squeeze I think in our industry as a whole okay first rabobank who are we of course ram bank is a cooperation we are an organization with which has members and we don't have shareholders this means that we're not aiming for optimized shareholder value but that we're actually investing in society and therefore we're we're not well we have a different goal than many other banks the result of this is that we used to be the only triple-a rated bank within the world that is that was not in government hands unfortunately due to the mortgage crisis that all changed but we're still one of the most solvable banks in the world so we have agricultural roots it started with a few farmers who wanted to help each other when crops went down or whatever so M we are within the Netherlands one of the largest IT organizations we have 3,500 plus I T employees that's our own personnel we also have do a lot of outsourcing which is about thousand people in India and if you look at our volume then the windows language iax problems is are no pretty large we next to this we also have a z/os and non stop which is out of scope in this story of course yeah so around 700 Java e observers this are not the focus of our story we have a challenge we need to change them fast that's and there's a reason for it and normally if I will if I'm giving this presentation then my usual question would be who's been inside a bank in the last month and around ten percent will raise his hands of the of the total audience in the if i start the following question ask the following question who's been inside the bank for something else that I'm overage then you see that all the hands are going down again because people only tend to go to a bank for the more complex products or the products where they need to give their signature real life if you look at this in a in time then you see that the where the physics to our offices has dramatically decreased our online contact moments with the amount of contacts with our customers have increased a lot and it's about a 10 fold if you look at today where the availability of mobile banking on every device leads to everybody checking it it's a account for about well once a week or twice a week this is a lot more than then used to be but on the other hand we are closing offices by the Dozen because nobody's coming there anymore so everything is available online if i look at google wallet google wallet is now insured by the federal deposit insurance corporation which means it's just a bank it's it's a payment accredited payment engine and lending building payment everything if you look at it it's all there and it's all in the cloud and if you see how Apple introduced I pay and became within one week America's launches mobile payment service provider then you see time to market we can only dream about if we want to we need to change and for our customers they're expecting these online services so we need to bring everything online which means that if you want a mortgage at rabobank tomorrow you won't be needing to give your signature in real life but we are using the digital signing which is for something in banking something that tends to like legislation be needing to being changed so our goal by the end of two thousand texting we want to have almost all our services virtualized online available for everyone who's needed and it's not only the payment but it's all our services also replacing your pass getting your credits done whatever everything you need and on the other hand given all the crisis is banking crisis we encountered the last decade we need to reduce costs in order to mitigate our risk the european central bank wants us to we need to and this means that we don't have the match the amount of money we use they have to innovate to develop new functionality and so we need to be creative and our customer on the other hand expects the surface which is available on a smartphone to be there all the time service disruption is not accepted where we we had a downtime five years ago of one of our data centers and nobody cared about it now if you have a down time of our payment engine or ideal engine for like five seconds twitter is starting to do to roar and and if we want to follow we need time to market it must improve dramatically because otherwise yeah we can't follow Apple and the way under skill that they are able to so we need to change be creative and change the way of thinking well income's continuing delivery in dev ops if you see look at our organization in the last five to seven years then we started with agile development we we started by changing the way we work after that we implemented continuous integration continuous delivery and now we are growing to the DevOps as it's being defined or described within under the internet or by the experts and problem is DevOps itself doesn't have a fixed definition it's more like a set of of outing of things you see and you can say it's DevOps so at the bottom we started agile by putting people together working in fixed sprints and changing our conversation from from scope so what can i get for ten dollars to what do you want so time and funding is fixed now and now our stories our conversations discussions and our whole mindset is about what can we deliver in of the next four weeks and how can we bring as much value in the next four weeks as our customers are expecting this means that we need to trust our specialists who are working on every day on the code on the applications on the maintenance of the applications and with our customers to to specify what is needed so it's not about command and control anymore it's about responsibility only ship and trusting your people and the team to deliver the last thing about our edgy our way of working is of course we're not using the waterfall anymore and delivering in about half a year we're delivering every four weeks problem was going live every four weeks was not we were not able to do that yet because we didn't have the means so this is agile this is a picture you can find on the end of the internet everywhere if you look at our process and you see how we grew with our agile way of working then the first complaint that that arrived was well we are delivering every four weeks but we can't deliver in production because there's too much administration for change management we there's too much manual configuration of hours of our environments and all scripts are stateful which means that if they work in a dev environment they are not working your production environment and the result was two releases a year although we deliver the code every four weeks and there is the effect of releasing twice a year is that we had a high risk and impact of every release example our CRM go lives used to have 70 projects delivering deploying in production at the same time which is something from a governance perspective you would not want and so are the normal reaction would be if it's risk full don't do it or do it as few as possible we needed to learn to change this way of thinking we needed to do it more in able to create a in order to create routine and to be calm well become a master in the process to be able to deliver more more often we needed to optimize your process so in comes continuous integration in which we automated the process from delivering code to creating a build version izing both the code in the build and deploying our software on our servers we used Excel deploy and it helped us to make the deployment mechanism for test the same as for acceptance products and pre-production it was in stateless environment which means that if I want to add another environment for deployment I can just use the same process and it will have the same effect the only problem is that the environments might differ slightly and therefore were not a blank canvas and it wasn't predictable what the application would do so I'll present the process is always the same which I said deployment is always the same it's push button delivered our application and the configuration of the infra for the application by the push of a button the software is placed in the process so if the process is always the same you don't need to discuss the process anymore because that's effort you put in the wrong place it's not creating value make the process of fixed process and focus your intention on the code and under your software there's almost no written documentation there is documentation about what's the new feature how could our customer use it or whatever but everything else is in the code even the documentation itself is in the code if you change the goal you change the documentation and there is no manual processes so which means this process is standardized and if we can do it on one the environment I can do it on the rest so that's nice but I still don't it's still just putting a stamp on an environment and stamping the application on it and and i'm done but i'm i'm not aware of how good I am at it I'm I don't have any knowledge about the time to market and there's no way of well I'm not in control yet so what we are building now and what we already realized that one of our environment is reputable infrastructure if i define it in a stateless way i can create a blank canvas for the application which is pre-configured for that specific application and it doesn't matter if it's in development test or production so i have reputable application deployment of course then if i can create environments on the repeatable basis and i can deploy on a repeatable basis then i can transport my code my builds but through the pipeline to production this is nice because then i have it's more like a real pipeline in it there you can create flow the last 1i want knowledge we want knowledge so if the test is not going well then I want to know where it broke who created god if it's happening more often what our time to market is how many iterations we have until we reach product production and why we're a time to market is better and in which project they're doing what to improve the time to market in order to learn not to judge but to learn so if we have a colleague who delivers a lot of code which feels a lot we know we can help him if
e have a colleague who create a lot of good code we know he can help the other colleague on the other hand if code doesn't break a lot then perhaps you should improve our tests so this is feedback is necessary to create learn and learning environments so what we're doing now is we're creating one standard it's one pipeline and for java if you the test itself is different per application it front-end may slightly different from back-end applications but if you but the process is the same and the process is automated so this is very now and what is what's delivered in the near future this is how we're doing it so we have a source control environment which is stash with at lashing it and we're creating bilson Jenkin and restoring them in a maven in nexus and from there we use excel release to transport our builds to our tests and to our acceptance environment load and stress test environment and in the end to production and in order to give feedback we are using splunk which in which we are at this moment creating a dashboard in which we can see our funnel so how many lines of code how can be delivered in production within for four weeks and how many business functionality is that how many how much business functionality how many story points are released every four weeks so we can learn again so this is what we're building now what it delivers for us is less weight waist we're not putting effort in the process anymore we're just building the process once we're improving the process or based on learning on the feedback loop but we are not everyone inventing our own process the effect lower costs less waste better automation due to we only the fact that we only have to automate on one process in one set of tools and therefore better time to market o our waiting time could be reduced so what it also delivered is our and this is something that's from a banking and the highly regulated environment is a bank is very much appreciated it gives us traceability so if go back to the previous slide then we can audit the what is in production and explain which wrote it all the code when and we can guarantee that the code we accepted the build we accepted to go to production is actually the build that's life and the application that's life so we can also guarantee that all the checks and balances were there to prevent that malicious code is brought life so that's traceability and socks compliancy from software control and distribution perspective and we have a nice explicit separation of responsibilities which means that there's there's no gray area between people which is there then a good basis for cooperation because you know what you can do with your own responsibilities you code and you know what your colleague is and what his or her responsibility is and from there you can support each other improve each other's results and help each other so this is better cooperation and elimination of human error which is again if it's predictable you can create the error in your development environment or your test environment if if once you see it you get the feedback you can improve improve and you don't need to have an error in production anymore so this is for what's in for us and its major the issues that need attention if when implementing a thing like this and it's all about change management the whole mindset of everyone in the organization needs to change first we need to think in products and results and not in processes anymore because the process is fixed we need to trust each other and to a change the way we bring things live we need to agree that documentation is not necessarily the best way to go documentation is needed but not like stacks of papers that nobody reads in that are well obsolete at the moment somebody's finished it and well this is something that just takes time onboarding is an investment in quality this is the sound like a no-brainer but an investment equality means it costs time and it means you need to think if you have the time with this moment we know you will go faster as if you want your own board but in the time your own boarding you're almost standing still there's a lot of discretion there's not a lot of change management needed and well there's time needed that you can't spend on your coat automating is letting go it means that you need to trust your colleague that he is allowed to do the same thing you did before because it's repeatable it's reputable infrastructure repeatable application deployment there's version management on everything so if it goes wrong you can roll back or roll forward and it doesn't matter so that means that if you see the picture of the food with a bullet in it it's the Dutch saying that somebody can shoot himself in the foot it's not a wise thing to do it hurts a lot but it won't kill you well so we from for my people they need it to allow the application departments to shoot themselves in the foot as many times as needed in order to learn if you if you're not making mistakes you won't learn a lot so going life requires faith and courage and therefore support of our the higher management if you want to go live every two hours or twice every two days then we need to trust on the process the funny thing is if you want to go live every two days you need to automate everything otherwise it it well it just takes to LA too much time and it helps you to eliminate all the mistakes in development environment so in an automatic automated environment you need another type of IT specialist and i discovered that just administrators are not the kind of guys you need everything is called my the infrastructure my department delivers is code the infrastructure of the application is menace in another way if we want to improve we need to think in pipelines not in an instant in production it's it's a different way of working so and you need therefore personnel that that is able to think in contact concepts and a conceptual improvements so repeatable repeatability is a key feature well I said that already everybody should be allowed to shoot himself in the food as many times as possible or as wished for as long as people learn doesn't it's it's the thing it's nice quantitative benefits this is from one project we and it's only the implementation of what we have right now which is application deployment and automation through the build server Jenkins the implementation we realized with exhale release is I don't have the the figures yet this is from one application so we have twelve hundred eighty eight hours which is almost one full time equivalent of a person and we didn't calculate the preparation time the time needed for documenting and making the the cookbook of going live which is adds up to around to a full-time equivalent figure we have around 100 projects all having the same potential benefits so there's a lot to gain here we have more control we have less we need less time we have we are better and improving and we're improving every day again because it's the feedback loop is there our ambition well we would like to expand on our log management functionality to imp and therefore increase the ability to learn because if we're able to learn we become more efficient better in control and well in the end it helps us to gain our competitive benefit we want to closely linked from code commit to application deployment it's there for one but I wanted as a as a a generic surface so it means that if one developer commits code it should be ready for production because if it's go through if it goes successfully through all our tests it will go live automatically this is what we would like and probably we will always have a green button for somebody responsible saying this is it has been tested the right way this is the right time go live now but in theory we would like to be able to do it all the way in an automated way so ambition is expand to net expand to power center and Oracle middleware and even simple CRM if we were if we were able to so this is something we are working on but also for symposium needs work what we've planned deployment triggers new virtual infra at this moment we we have for one environment that we can give on demand Taylor the infrastructure on a standardized basis but at this moment we are developing a generic service for it where we can deliver tailor-made definitions of applications within minutes and it's something that's necessary for time to market but also for cost because every virtual machine that I'm running is costing me money so we wanted to be able to deliver to an external cloud provider because we want to use our don't want to pay for x86 hardware if we can take it for ten cents in the cloud for the period of a week so my dream is fully automated pipeline which means a developer if we start the project we have a place a source repository for code for configure infra a configuration and for test code and we supply our developers and tests personnel with interfaces to their tools so from the code the environment will be managed it's because it's just a part of the software configuration whom I not that also means no accounts on our infrastructure nobody logging in anymore because all the data is in feedback loop and the deployment mechanism and the pipeline are fully automated this will take some time though so let's I'll give it a year and I hope we're there but it's a lot of discussion because it's all about having faith and also believing that it will work yep thank you very much sounder that wizard well I've been fascinating stuff and I have to say obviously I had the benefit of having a bit of a sneak preview of what's done that was going to talk about it is Andrew Phillips here again some of the things that especially stood ask for me and going through this material and I think they're incredibly relevant to the challenges you know a lot of our users and customers are facing but i'm sure a lot of you in the audience as well is first of all anything can i have some kind of confidence that i will get benefit out of this even if i don't go straight to the end yes under finish talking about this dream and it's clear that blog post that you read about etsy deploying five trillion times of a second you don't have to get all the way there in order to get significant benefits out of getting started with these initiatives but of course there's no free lunch as asunder pointed out very honestly and truthfully this takes some initial time right you this doesn't just happen okay well we'll take half an hour every day at the end of the lunch break in to admit this happen there has to be some kind of buying for this but the more demonstrated proof there is that this can work and it can work in environments that look quite a lot like yours I think the better the case becomes for you to take that you know as as something that you we have confidence that we can get this done and a discussion would sound a little bit early and maybe he will touch on that in the questions or so about know what the structure is around the who's the owner of this initiative what is required in terms of executive buying or sponsorship in order to make this happen and clearly if you want to do this broadly and an admirable turn in that kind of environment I think we know that some nowadays at some level of executive sponsorship is required you can start these things small you know five motivated people in that Tiger team with a couple of service and they found somewhere but once you to make this big and you started touch some of the more business-critical applications there has to be fine this is worthwhile I don't want to steal too much time here from from the questions because I'd really like to get into some kind of discussion but a couple of more things that jumped out at me that I enjoy very much the principles behind sleepy and elves don't just work for that the Roku marketing web ad letter that you wrote you know there's some dimensions some technologies that really count as heavyweight big enterprise stuff and I think there's been a tendency in the pasta sort of pretend they don't exist well let's do that also the micro circuits stuff over there that we're building brand new and we can take 10,000 new frameworks off the shelf we won't have any of these problems of reality and of course guess water and I'm a discussion with gene came upon this reasoning like DevOps in the dark corners is read or write in order to that and you can microservice that shiny thing that you're building eventually we'll come back and it will call your any frame there's there's no way around this so you have to in terms of the scope of what you're going to address you have to bear in mind that this will touch things like powers and or acquittal where you're you know legacy or enterprise or whatever you want to call them java stacks your dotnet stock drops christian you shine is josh and to see demonstrated proof that these principles can be applied to those kinds of stacks and it works really well do this with existing applications and if you look is really significant benefits i think is very motivating comforting are we one foot um and yeah i think i think one final thing that the notion that this is all that everybody says asked a better cheaper that's it now who would say no to that kind of story obviously nobody would it needs but people know what do i do now I think that's understood some very interesting and concrete points about waving that I started in which particular technical challenges you need to address my deployments good then you need to think about your environment because even if your deployments repeatable if they're on a shifting sand environment that will be great and that you beyond that now you add you take away access to those environments by making them all version beautiful its separate separate back to the point where you have that that dream goal that fully automated pipeline and and now I'll do the sort of two minutes about why of course what we try to do helping with that and for those of you who are not familiar with our technology standard mention two of them i'll briefly sketch them out there's plenty more information obviously out on our site I'm very happy to give you as many demos and discussions as you like if you'd like to learn more about them so XO release is our kind of end-to-end platteville orchestrator that son that was mentioning as the thing that allows you to look at all the work that needs to be done from the moment that you've completed development all the way through the when your code is actually lie and of course there's a bunch of orchestrators out there but what we think is kind of unique or what we try to enable with this tool that you can get started with clever process you have today and the reality in most organizations is that certainly as you get towards that last moment like bit towards perfection not everything is automated yet you need to interface not just the technical tools but also the process layer your change management tooling service or incident management to ignore all that kind of stuff and the people that go with that the project manager to the extent they still exist or maybe your business users you need to have an interface that is relevant for them we obviously be dogged with our own tools so we run our own product releases through expertise and did the reality is yeah of course we have the same goal we want to push every commit out to some area where you could download it but as sundar said your documentation we want to write release notes we want to write updates blog post so there's always still some manual component and rather than pretending the data component doesn't exist and sort of tracking that in an Excel spreadsheet that we sort of secretly keep shamefully we actually make it part of our pipeline because it's real it's in there we need to know we need to have feedback if it turns out that publishing that blog post is the thing that always holds up our release then we have the data that allows us to say okay we have to go fix it either we get rid of it or we find a way to do it better or whatever it is Excel deploying well son there already mentioned that that's our that's the deployment thing that's like you have applications probably a ton of them you ha
e a lot of different target platforms whether it's the job as the bell nets the power centers they matticus and big data tooling so not just web apps but all the stuff that you have running in your environment you need one consistent way or maybe you don't need but it's certainly very advantageous to have one consistent way to package and deploy your applications out that in enormous diversity of target systems because then you only need to think about one reporting model one audit trail one security control set up all those overarching concerns that cut across all those different technologies you don't want to have to reinvent that wheel all the time with every different technology so what we try to do there if you dine capability with one tool that yeah has that platform interface between your apps and where they need to run and the last one is test view so I'll just find a quick minute on that sundar wasn't talking about that really what we've seen but but he talked about know the notion of quality and feedback has been central to the confusion process and we obviously believe in that very strongly and so to some extent we've been a bit surprised by the fact that a lot of the discussion around DevOps and CD tooling focus is very much on the automation side of things like how can I spin up servers automatically how can I run containers automatically whatever it is and not so much on how do I get feedback out of this running system as it actively and efficiently as possible because without that information that quality data you can't really make a confident decision about whether to go live or not and then in the worst case you get a knight capital or a new york stock exchange and of course that has tremendous negative impact on your ability to do CD because then people go back in their bunker and say oh no we need to introduce these big processes to prevent bad stuff from ever happening and so we'll have like three weeks of manual whatever testing and you're back to the increased risk of trying to deploy everything at once so quality really needs to be a central element in your thinking of your planning of the process and yeah the more quality data you have the better obviously the challenge becomes a little bit the more test tools you have because that's really what quality boils down to the harder it becomes to aggregate all this data together so just in the same way Sumner was talking about splunk as giving you that essential information about what systems are doing at that level and obviously what the splunk do it aggregates all that data no matter where it's coming from gives it to you in one single pane of glass XL test view does the same for all your test results so we don't care where the test results are coming from selenium gattling Vanessa cucumber and there's a true team tools out there this is about bringing the data together and showing it to you in the way that you can configure them that makes sense for whatever you need to whatever questions you need to answer so then ultimately when you hit that big green button that summer was talking about do we go live or do we not go live you have confidence that you're getting it right okay enough about that enough about meeting us let's open it up to questions really looking forward to continuing this discussion fantastic thank you Andrew and you know what sonder thank you just real quickly because we need we have a lot of questions here but the the insight into banking and because people always say our DevOps is great for startups are great for google and facebook but does it help you know organizations that have compliance issues that are you know in the finance sector and understanding how banking today you know that the the it's moving to the web it's moving to mobile application see the branches themselves are being shuttered down and and how you guys are I do accomplishing this using DevOps is it's just huge we don't get that kind of insight so thank you so much let's get to some question though so first question here is what is the role project managers of playing in your model Sanders and are they helping create and maintain the feedback metrics how do they call out collaborate with scrum masters well the the role of the project managers is changing where before it would be the first person you talk to if you want to speak about the the content of the project it's now the scrum master and the project manager is mainly responsible for having the right people and the right amount of people in the projects having the having the process is running and facilitating everyone that the whole scrum way of working is is well it's doing a right job is it's gaining value so it's it's much different than before okay and just a quick reminder folks there is the question section of the GoToWebinar control panel and if you do have questions you could put them in there we will try to get to them or will try to respond in writing after the fact next question and this probably can be an Andrew question as well is did you have to transition from a blame culture in order to allow for learning and mistakes to go for a summer maybe you first yeah it's something we're still struggling with it's difficult yes we need to go from blaming culture to to a learning culture but it's very easy to blame so this is something we're working on and the the surprising thing is feedback helps because if you know who's if there's optimal transparency then you can have the conversation about it but then there's the very important role for managers to facilitate the process and facilitating the the migration to to learning instead of blaming and I think maybe just to add to that I think 11 so an interesting story that Gary Gruber the guy who did the you may not only HP transformation lay their famous their software thing that features in the Lean Enterprise spoke very strongly me hold an interesting story about one of the things about feedback being no blame is it obviously it's very common if things fail a lot like because then you're under pressure and then so the tendency is not to figure out how to dig yourself out of the hole but to try to make sure you're not stuck at the bottom of the hole but I think one of the things that that he was pointing out is that what fast feedback especially does is it allows you to to localize smaller errors that people don't feel that ashamed about right there's a difference between pointing out that your commit broke the code in the dev environment and if you know sort of you know and we all know from teaching children right or animals or whatever you have to give the feedback quickly there's no point and waiting six months until everything blows up in production and then saying ok so the dev team clearly screwed up because the code bro hey that's a incredibly broad blame I mean there's no way you can go to one particular person say look I think you made a bad commit how do we fix this for next time because you don't know who did it so the blame gets sort of spread incredibly broadly and if you have a fast feedback cycle a you catch problems earlier so there's less of a need to go and like hang somebody out to dry plus your quote-unquote blaming you're trying to learn about a smaller problem so it's also less of a like okay well you're going to be fired right now because you've made a bad commit um and so I think that that teaching in the same way as you reduce risk by delivering lesson doing that more frequently you can also reduce blame I breaking in smaller bits and doing that and break fast as as soon as possible so you want to have the first test to be as discriminative as possible in order to to now what know where the problem is as fast as possible excellent you agree get to know okay next question is can you um can you share more about how you are able to satisfy your regulatory requirements while doing continuous delivery yeah of course well the main subject is software control and distribution which is being able to guarantee that the code that's running in your life production environment is actually the code that you intended to be there and so you can only guarantee that it's the right code if you know how the code came there how we were able to to deploy this offer so if you have manual steps in your deployment process the first moment the first manual action makes it confuse you don't know anymore you can't guarantee that the one the person that's locked in did something wrong or they did something malicious something you didn't want to happen so therefore by closing the loop between our source repository our build server and our deployment mechanism you can guarantee that the code that the bill to bring you you brought life consisted of code you approved up got it excellent what about database releases how do you recommend these are released in the continuous way the new version of a database all right sorry I'm very odd name yes I vocation in education with a with a database facility because in the last one you can use at this moment transaction after roll back and before we also used previously Oracle snapshots for 40 back so if something went wrong we could go back to the snapshot and the situation now is we can just push the button roll back and exhale deploy man this is for us that makes it that makes it easy yeah just a quick reminder these slides and the audio they're actually the entire presentation will be available within 24 hours via mp4 on will be on dev ops calm and then separately the slides as well though the mp4 include slides next next question this approach clearly needs a lot of buy-ins from various levels in the corporate hierarchy you bet did you encourage various levels of resistance did you encounter excuse me did you encounter various levels of resistance along the way for example some form of Y changing something that has been working for the past 10 20 30 years if so how did you work your way through Oh with a lot of effort and problems and struggling to be honest and the thing is my our CIO really believes in DevOps now so since a few months everything has become a lot easier we have a Java guilt is Spotify is calling it a guilt but in the in dash is called the up and those are now allowed to decide what our pipeline should look like so which tool should be there who should be able to do what and this is something new before I had one application owner who said i have a problem can you help me yes i could of course I but let me black by a application deployment mechanism for you and from that single project we expanded to a next one who had their project manager was who delivered who totally believed in continuous delivery and who was willing to invest time from his project to improve and then from there on we just were like telling the story and it's just selling the internally selling the project product until our CIO well started to become interested in DevOps so it takes it really takes a lot of effort or you need to have one person hi in management who believes in the subject ok I am that that's very true and you know this is a subject that we've actually dealt with on other webinars and ones we've written stories about here at DevOps calm and that is do you do how do you do DevOps is it top-down bottom-up it's a little above and how do you how do you get executive sponsorship for it and and I think the answer is it doesn't happen overnight and it doesn't happen without some stops and starts and you need to build on successes and I imagine sander that that's kind of what your experience has been from hearing what you went through yeah and it in the ended it the the most effective way was just inviting an expert who was working with our methodology and just show it take the tool show how a deployment works and show how the the pipeline from codes to production how easy it can be if you're able to standardize and automate so from make it easy oh if I if I can also just jump in your briefly preferable I think that one of the one of the things that of course you mentioned at the beginning of Sanders said it as well there is just now we're now in a DevOps means everything to all people right it's the new cloud from a terminology perspective so it can also be a challenging label to stick on something because it's like so what are you trying to do here the reality ultimately is and that the real initiative is called speed stuff up without breaking everything right and what whatever you call it I don't think makes that much difference that the problem is well not very few people are going to disagree that speeding stuff up is a good idea that's always true i have been in some discussions that surprised me at first as well where people didn't eat candidly say no we released twice a year we can go any faster than that anyway because you know our customers won't do it or whatever and and the reality is yes this is not a silver bullet if you don't have a need to do it if there's no driver to do it you're not going to get it off the ground especially if things are have worked reasonably well the fact of the mat is most companies do have a driver somewhere and it ultimately just a question of you know are you going to be a little bit proactive about this whether that's in the context of an individual project which you really need to get out of the door whether that's as half of Uncas doing look at the way the market is developing and stuff is going to happen whether Apple bunk does anything or not so I mean you can wait and wait for your competitors and there was an interesting data point the last thing I'm going to say on this I think ten years ago the cost of starting a tech startup to compete with anyone was about five million votes and the cost today is five thousand dollars so i'll never be exact statistics behind that but if you think about that like a factor of a thousand decrease in the difficulty in competing with your business if that doesn't persuade somebody at the strategic level to think okay well if we're in a challenge in a market where we can be challenged there's any more people who can challenge us today and I don't know what will agreed and during that that's very much true today and it used to be that a software company would cost forty million dollars and today you can get ass ass out the door at a fraction of the cost we have time for one more question and then we're going to need a quick answer on it but it's what makes CD Projekt successful mindset collaboration or rather tools and framework and you're going to let you go first and then I'm gonna give sander the last word I'm going to be very quick on this I mean of course ignoring my bias I think Carrie Kruger and it's but it gets it about right it's somewhere between 70 30 to 60 44 process culture versus the tooling I mean the tooling is it's not so much the tooling that drives it it's more that ultimately acceleration requires removal of slow processes and you need to have reliable things in its place so automation is somewhat a given because there's very few ways of doing this without automation but the thing that makes that that tooling successful and they gives you that level of visibility and ultimate confidence that we can use the tooling to its full potential that's a people process thing absolutely just real quickly Gary groover does have a new book coming out on IT revolution press and we should be on the lookout for it Sandra what about you well I actually totally agree so the tools is just a necessity in order to work and deliver faster it's not that well we have the tools let's make it successful and we don't change the way we work its culture its mindset it it's the team that makes it a success and you need aught emotion for it and fortunately there's companies like CBL apps who deliver tools that assist but it's always the teams excellent guys we're one minute over so we're gonna have to call it a wrap Andrew Phillips thank you so much for your articles on DevOps com for participating in today's call for all you do in the DevOps community sander edameh of rabobank thank you for this incredible insight it's a real world example of how a banks are using continuous delivery and DevOps and to our listeners thank you for joining this is Alan shemal for DevOps com thank
ou and have a great day
