Streamline pipeline bid management for research and development

How to create outlook signature

[Music] welcome to the SEI podcast series a production of the Carnegie Mellon University software engineering Institute the SEI is a federally funded research and development center sponsored by the U.S Department of Defense a transcript of today's podcast is posted on the SEI website at sei.cmu.edu podcasts Welcome to the seip series my name is Suzanne Miller and I'm a principal researcher on the SEI software Solutions division today I am joined by my colleagues and Friends Bill Nichols and Julie Cohen they are also members of the software Solutions division today we're here to talk about their recent work looking at actionable data that can be pulled from the devsecops pipeline welcome Julian Bill well thank you glad to be here so this is going to be a fun topic but before we get into that topic let's begin by having you tell us a little bit about yourselves what brought you to the SEI and the work you do here what's cool about it um Julie you're new to our podcast Series so why don't we have you start so my name is Julie Cohen I've been at the SEI about 20 years now before that I was in the Air Force for 20 years as an engineer and a program manager with a lot of different jobs when I left the Air Force I had a couple of jobs in industry and then I joined the SEI where I've worked space programs and and um non-space programs acquisition engineering um it's it's been a been a good ride what's the most fun about what you do actually helping the customer seeing the customers struggling with something uh helping them to see different options and then watching them pick a good option and move forward I predicted that answer I've known you for a while um we both have that same same sensibility Bill uh tell us tell our audience about yourself those that have not run into you before and same thing what's cool about what you do here okay well my background is uh experimental particle physics that's where I actually got my PhD but I spent the next 15 years or so working at a naval nuclear lab doing the programming for scientific and engineering design and Analysis tools so the I wrote things that were used to design the Next Generation nuclear reactors to build simulators and so forth as we went into uh not only developing but modernizing that we got really involved in modernizing a large suite of analysis software and that's how I got into software project management and it was also a good exposure to some of the needs of the Navy as far as being able to meet certain types of schedule demands for example you had to have simulators ready for your crew to train on and the software better be ready for that you had to have the onboard software ready for uh the crew when you're ready to launch the boat you know the schedules really did matter not as much maybe if you're a NASA and you have to meet a launch window because there it's going to be a couple years before you can go to Mars again but the schedules were important well we worked with one of the software one of the teams from the software engineering Institute on how to do some of this program planning and that's how I really got into it and after a few years I actually came to the SEI to learn a little more about that and to share what I was learning with other people and to find out what other what others in the world were doing see see this from broader perspective so I've been with the SEI now since about 2006 so it's been essentially half of my working career what I find most interesting is you know I really get excited when I get a hold of the data and can try to make make sense of it understand what it means think about how you can improve the process or make predictions yeah it's it's like wow they actually pay me to solve these uh puzzles yes you are a puzzle solver um and and I think all of us are in different ways um but I I want to sort of move us in into talking about devsecops and just before we get into your topic I want to remind our audience members especially those that are new that the SEI has lots of resources um including podcasts that will provide you with an overview of devsecops we have things from uh The Continuous development uh deployment of capability directorate we have things from cert we have things from uh your group so we have lots and lots of things and so we're not going to go into an overview of what is devsecops here we'll have links in our transcript to all kinds of good resources but I do want to talk about how all devsecops has changed the game when we are talking about the challenges that program face and why automation is so much more important today than it might have been 20 years ago or 15 years ago when when you were doing you know some of the analysis that you were doing in the past so um I don't know who wants to take that one first but well let me take a couple of things on this one uh first of all devsecops is all about continuous integration continuous deployment infrastructure is code automation of tests and builds and deployment everything is automated so if you can automate these things it means you have to be able to describe your workflows in your process with a fairly high amount of precision which is what I've been advocating for years but there was always this resistance in the software Community to doing that but there's also been that counter tendency if people love the tools they want the tools to help them well devsecops the tools are doing a lot of that men will work with pushing things through deployment managing the system and what's really changed with this level of automation is the Precision with which you can actually describe your workflow in your work process and that provides a lot of opportunities for measurement now the challenge is because things are coming through continuously you're doing this continuous integration continuous deployment you have this constant stream of data so just keeping up with the data becomes a challenge but everything is being driven by tools well the tools should be able to help you automate that data collection uh that presents another set of problems but the tools then which are enabling this fast rapid development also provide you an opportunity to measure what is in effect a precisely defined workflow and that's what's really changed the game in the last few years this realization that we do have a repeatable prescriptive type of process that we can measure and evaluate and you know I I know some of the work that you did earlier that resistance to defining a process in in precise ways that can be measured I know from the work that I did in that time frame a lot of that resistance came from if I Define it then somebody's going to want me to collect measures for it and the collection of measures in that time frame was not automated you know I remember us having screen little screen things in the you know upper left-hand corner just to record time stop and start timers and things like that that were actually intrusive you know to the flow of the work which is one of the reasons that we would see some of that resistance so the fact that we are 15 20 years later able to make Process Measurement blow drag is is something that's not just significant for the developers because I'm going to go to Julie now the managers have always wanted to be able to have the kind of insight that we now can have because we are able to collect the data systematically because when we're collecting it manual manually it's not systematic I I'll raise my hand saying you know I've tried several times in different settings to collect data manually and you know I wouldn't pass my polygraph if I were to assert that I consistently collected my data you know we're humans right so tell me Julie about what is it that the program managers that you work with in the acquisition settings what is exciting about being able to get this work for that role in the in the development process so I think one of the things that's probably the most exciting is having more more access to real-time data or at least near real-time data for those of you who have dealt with earned value Management in the past right you get a report that ended the the end of the month but you don't get that report until the end of the next month and so by the time you get it the data is almost 60 days late in some cases very hard to manage with data that's that late even a regular cost report can be very late and so while you know you may not get precise cost data from the devsecops pipeline you can certainly see effort and how much effort is being expended and those types types of things in near real time certainly your schedule information is much more real time and you can see what was planned and then what was actually accomplished in much closer to the time when it's being done and so if problems start to pop up you you know much sooner and you know that was always the premise of earned value anyway was that once you start to get behind you can never catch up so the earlier you know that you're behind the better off you are and and now you you can probably find that out even earlier um same thing with product performance you can see those tests and you can and and as far as risk goes you're lowering risk because you're automating those tests you're redoing those regression tests with every build and so that lowers your overall risk as well so those are some of the things that the program managers um really will benefit from and you know I we've all been in the position of having that data that has a time lag to it that makes it I won't say useless but you certainly can't take the actions that you would want to take if I'd have known this two months ago I would have taken it I would have made a different decision and that's I think to my mind that's what it comes down to is enabling program managers and acquisition people in different roles to make better decisions if they have real data that's near real time that is that they have faith in then they can make decisions more confidently and they're going to make better decisions now all of this of course depends on tooling right and um I think that one of the challenges I've run into in areas where we're trying to use devsecops and productive ways both for developers testers and managers is that not all tooling is created equal and oh by the way the tooling that the development contractor has and the tooling that the test lab has and the tooling that the government oversight groups have are not always the same and not always compatible so can we talk a little bit about sort of what are some of the key tools and types of data and are there any insights that you've seen that you've you've had in terms of you know if you could just make this piece of the pipeline the same tool that would make so much difference I don't care about all these others you know we're never going to get everybody on the same so there is not one tool to rule them all but you know if we at least get this kind of tool compatible if not the same then things would be much easier I don't know the bill did you want to take that one you're smiling yeah well let's look at it from this uh point of view the specific tool doesn't matter so much as long as you have the right interface to get the kind of information from this tool that you really want so whether you're using the open source version of Jenkins or one of the proprietaries the question is what information do you really need to collect and I think the biggest rookie mistake in most of these efforts is they start off with Hey What Can we measure and that's always the wrong thing to do for a variety of reasons but the I think the biggest is you end up measuring a lot of things that you just can't use and the second problem is not only if you measured a bunch of things that you can't use but you've almost certainly missed measuring something that you needed in order to make the measurements you did take useful so one of the things that we've been finding is that you can get a lot of great information from these tools like the Jenkins is going to be able to tell you things about uh whether the bills were successful whether you when you push these things into deployment from your git lab or whatever revision control you can tell things like how many uh how many interfaces were changed or how many files were touched what went into a specific build but but how do you What did they build you just know that a certain amount of code went into a build it passed some tests it went into production okay what went into production how does that Advance my project unless you are thinking ahead to some of the problems like how am I going to tie this into the overall workflow uh with something like what's going on with my git lab or my jira what are the tickets how do the tickets roll up into features and those features how do they roll up into capabilities if you haven't connected those different pieces you're just getting a bunch of Point information that isn't really going to be helpful for the programmatics uh it reminds me of one of those old jokes here's a partial score Stanford 24. perspective that I hear a lot you know can't we just all have one tool or can't we have this tool but I I really appreciate your point that in today's world it's about the interfaces we've made so many so many strides in essentially externalizing interfaces so that it is feasible for us to have compatible interfaces across lots of different tool sets that that should not the the lack of um of sameness in the tool chain should not be a barrier to us being able to get the data we need but it means as you said we've got we've got to think about what is the data that we need now I want to sort of go to Julie and say what are some of the types of data that program managers need at various points throughout the software development life cycle that our audience may not be aware of as being valuable that can we we can pull from these kinds of of defsecops environments right so probably one of the best things is scheduled type of information where again you've planned work you know what you plan for that Sprint or that increment and you can now see um very clearly what's getting done and what isn't getting done and so if work isn't getting accomplished if there's a capability that is having problems and lagging um you know in the next increment or the next Sprint depending on how you're running your program do you want to prioritize something else and and think about that capability a little bit more and what's needed um another thing is quality right you can you can run code checking tools that can help keep track of quality uh so that perhaps you're getting less technical debt um from the day that you deploy you know there's always some technical debt but if you're checking your code regularly then you could be more clear about what the what the quality in that code is um same thing with cyber security right that that is becoming more and more of an issue and these automated tools can help you get a handle on just where you are with cyber security and and where your risks might be um the other thing that it helps you look at is stability right you have a stable process do you have a stable pipeline or is your flow good are you are you meeting your your velocity on a regular basis those types of things are things working well because if you've got a pipeline that is fluctuating up and down and it's always something different then you probably need to do something so some of these early indicators um you know maybe not for the program lead but for for a manager who's who's involved in the software development they can get some some very good metrics um quickly enough to enable them to make some changes if they're needed so so I'm gonna go uh on both the points that you both just made um I do want to remind our viewers and especially Bill Nichols both of you um are involved in this work you know we do have a set of um techniques called gqim goal question indicator metric that are actually designed to help you figure out what are the things that we really want to know about and we can get into timings we can get into Fidelity but um I think that that point of just because you can measure doesn't mean you should right because even with automation it costs money to gather data as you were saying earlier bill it takes you know it takes time and effort to manage the data to keep it fresh to understand what is stale what is not stale and what is relevant what is not relevant so you know there's a whole uh a whole system of measurement and Analysis that you need to to deal with SEMA is software engineering measurement analysis that's the group that bill Works in um you know there's a whole lot of things that go into this besides just having access to the data so um to that point Bill maybe or well Bill and then Julie um what are parts of the system's life cycle where data captured from devsecops pipelines you would say you know what maybe that's useful for some Niche over here but it's not going to be generally useful data especially to the management are there some types of data like that that you want to highlight to our audience well I think the from a program management standpoint the kinds of things you're most interested in are things like what are the costs and what is the actual buildup so I don't a manager shouldn't care about the story points how many story points you're getting uh the velocity totally useless what you want to know is how is this Burning Down uh my uh my product increment backlog and what we're finding is even among the organizations that really have the rack together that are running pretty decent pipelines they'll get into the situation they'll be uh two months into their three-month product increment and they still don't know if they're going to get all their features in well that shouldn't happen that but that happens because they don't have they haven't thought ahead of time what it is they need to measure they're so focused on individual Atomic things like story lead times that they've forgotten how are they actually accomplishing how are they aggregating these things how are they aggregating to the actual work uh and the other problem is you've got to deal with this between teams a lot of the work a lot of the measurement that we're doing in the agile world is idiosyncratic to a specific team and you have to have certain ways of normalizing that work and unless you've put some thought up front into what you think these different components are going to cost you have no no basis for comparisons right so that's one of the things that we are definitely finding that there hasn't been a lot of thought to how you do some of these upfront estimates how you trace these through the system how you update the estimates because you're going to get different estimates when you start going through the development Pipeline and you get the bottom up estimates as opposed to Management's top down you've got to be able to track those and you better not be throwing these just into any old data Lake because I'm from the school that says you really have to think about what you're measuring because if you have to solve that puzzle afterwards it's going to be a real heavy lift and if you want real-time data well that's not what data Lakes are for those are for handing it off to some researcher who has grad students and time on his hands but if you want an answer today you better know what that data is and I I tend to most of the data Lakes I run into are really data swamps because they they take you down you know they suck you in and you end up trying to deal with data that you end up saying why did I do that that's not yeah I call that the software and the software analysis measurement pool yeah yeah the swamp oh oh oh very good very good Julie are there any particular types of data that you've run into where you go oh please don't collect that that's just not going to give you any help at all well I think I think Bill hit the nail on the head with you know lower level velocity and day-to-day types of things um one of the things I will say though about the research that that bill is doing that's really important is looking at multiple pipelines so if you have a big project chances are that you have more than one Pipeline and the things that are on Pipeline one may need to interface with something that's on Pipeline two and there's a date where one set of work is needed to enable the other set of work and so being able to look at um information across the Pipelines to know how that integration is going to happen is really vitally important we know that that integrating software is important to do early and often um devsecops in general help helps that tremendously but if you have more than one pipeline you need to understand what data is in each one and how they're going to come together so how is that work progressing in each of those at a top level is very very important to the program manager and I don't know about your experience but my experience when you have that multiple pipeline situation is that everyone is you get this local optimization right versus systematic systemic optimization everybody's trying to optimize how the workflow and the measurement and everything works in their Pipeline and and it's easy for them to forget about who they need to integrate with and look at you know I may need to slow down a little bit because I'm not really going to get a chance to even integrate with this other peace you know until this this other time period maybe I need to be working on something else while they're catching up you know to where we are so so so this is very reminiscent of how we were dealing with uh you know manual processes right in the 80s of who's catching up to whom and when are we going to integrate and where are we going to integrate but we were doing it from a manual Viewpoint and so in some ways we had more time because it took longer to get to an integration point and we weren't as fixed on uh incremental iterative fast iterations of small batches we were doing you know sort of larger components so you you had actually a little bit of slack a lot of times to get to that point what we're seeing I think here is we're moving fast enough that those integration disconnects are immediately visible and it's it's you know it's on it behooves us to look at that whole system of pipelines is that the direction that your work is going bill because I I I have to ask you the question what are you going to do after you you publish the things you're working on now well there certainly is some of that but let me let me take what you've been suggesting and build on it a little uh you're probably familiar with the way they recommend doing things in scale natural framework uh that's where you have they use the Release Train metaphor basically all these pipelines are going flat out and whatever's ready you integrate and you put on and that's fine that's actually the way you should be doing most things as far as you can but but in our world the dod where you've got these different contractors there are going to be dependencies in those Integrations and you can't just make those go away by waving your hands and that is where this uh this problem really comes in uh that's why this multiple Pipeline and the dependencies is especially a problem in the dod world where you have real products that are coming together and uh you better have all of them or the Warfighter is not going to be properly equipped for uh for the adversary it's very different than you get in a lot of the commercial spaces so I think that's why it hasn't been addressed uh addressed previously because it's just not a problem that is as widespread in the commercial world now the kinds of things that I'm interested in looking at is extending more of this work in digital engineering the digital engineering hearing world is uh is uh really exploding but they're running into all of these other Growing Pains they've got all of these issues with the different uh portions of the work cycle don't work with the next that they don't have the proper interfaces between them and they haven't caught up with how does this affect your workflows which reminds me the another thing that we could have touched on in the tooling discussion is well even if you don't have the same even if you have a lot of flexibility in the tools even if you go back and say we're going to use the same tools that doesn't mean you're necessarily going to be using the same workflows or using the same kind of accounting systems in the workflows so you have to think through those issues how you're going to do the accounting and that comes back again toward gqym discussion what are your information needs it really helps to think ahead of time what information am I going to need to make decisions and that should inform not only how I instrument my process but how I do the bookkeeping the accounting and the analysis in order to get the information I need from it so I think that's going to be uh I think that's going to be very interesting for at least the next decade as we continue to move into the digital engineering World more and more of the work is going to be represented in these digital artifacts and you know I I smiled when you mentioned that because that is one of the areas that um the adult transformation team is very active in right now in terms of understanding how does digital Engineering in within an Agile development process and how do we apply agile within digital engineering to get better results you know get small batches out of that so this is definitely an area that's active and measurement as you know is one of the it's it's like the last thing people want to talk about they wanted to talk about how do I get something and and that has to be uh you know so we have to we have to we have to nudge people in in a good direction Julie what's next for you in relationship to this work what's the things that you're hoping to to achieve As you move forward with this work so I'm really looking forward to the piloting uh phase where we hope to actually use this on some programs and looking at what kind of dashboards are available and um showing those to PMS both of the programs that we're working with and other programs to say you know what what could you do with this data if you had a dashboard like this and what would you rather see and you know what data is most useful to you in this type of uh in this type of an environment and getting some really good feedback as to how well this can help the program managers and I take Bill's point about it's not just having the same tools and even the workflows but doing your accounting and bookkeeping the same way I can't even tell you Julie you and I both run into this where we think we're getting hours we think we're getting effort hours and we're really getting calendar hours and it's like well that doesn't help me that doesn't tell me what it takes to build this it just tells me how long it's in the cycle and you know something as simple as that can just completely throw off your analysis so I feel like we're in a Back to the Future kind of place because certainly 20 years ago there was a lot of discussion uh pssm the you know practical software measurement trying to get people to get clarity on defining what your measures are the seis work in the mid 90s on trying to get clarity I mean the the things that we did all these templates full of here's all the data that you need to understand is my data the same as your data we're coming back around to it but whereas before people would look at that stuff and go oh dear if I do this then that's just gonna be so much work for me to gather and figure out and how am I going to analyze it my Excel spreadsheet isn't going to handle this now we actually have hope of using that Clarity to actually get some interesting and relevant information that as we say in the title of the of the of the podcast is actionable and that's what's exciting to me is getting to actionable data uh for for lots of different roles I mean we can have a whole different conversation about how this leads to actionable data for test Engineers for example but but I agree that that program managers are have been left out in a lot of ways of the conversation about what to do with information that comes out of the devsecops pipeline so I'm really glad that you guys are working on this and look forward to some of our other teams working together with you because we all have different pieces that that you know we know about in relationship to the devsec optional engineering and all that kind of stuff so um I want to thank you for joining us and take talking with us today this has been a lovely conversation for me as as people can probably tell I have some passion for these things and I have passions for many things that's why I love doing the interviews but I I do want to remind our audience that we will have lots of links in this transcript to resources that we mentioned and that are related there is a blog post coming out on this topic so uh you'll want to look for that and we want to remind you of the audience that you should be able to find this anywhere that you get your podcast if you like SoundCloud Stitcher Google and of course my favorite the seis YouTube channel um so please feel free to see us there or wherever you want to get your podcast and I want to thank you again for joining us today [Music] thanks for joining us this episode is available where you download podcasts including SoundCloud Stitcher tune in radio Google podcasts and apple podcasts it is also available on the sci website at sci.cmu.edu podcasts and the sei's YouTube channel this copyrighted work is made available through the software engineering Institute a federally funded research and development center sponsored by the US Department of Defense for more information about the SEI and this work please visit .sci.cmu.edu as always if you have any questions please don't hesitate to email us at info sei.cmu.edu thank you [Music] thank you