Collaborate on Expense Bill Format for Quality Assurance with Ease Using airSlate SignNow

Expense bill format for Quality Assurance

[Music] hello everybody welcome to the business performance podcast I'm Henri Schneider and my guest today is Bill Curtis who has been a longtime friend a colleague of mine he goes way back with the quality movement leading the effort in the early days with the software engineering Institute and watched some free to build the original capability maturity model he is also instrumental with the people mature T model he was one of the principals of one of the premier consulting companies Terra quest metrics and lately he's been involved in developing international standards for measurement of size and quality of software so please welcome Bill Curtis glad to be here okay so bill would you please just to get everybody familiar with you who don't know you this describe what you do with cast right now your customers and the types of services you're providing in your position well cast is a technology company we have a technology that analyzes large software systems multi language multi-layer software says to find all the weaknesses in in the code both in the architecture and the code itself so it analyzed the security holes and things that will crash the system things that'll make it hard to maintain and then provides that feedback and some metrics based on it some metrics it would tie to the risk of the system back to the back to the customer of the technology so as part of that when I was hired the CEO said look at you know right now all our measures are proprietary and we know there's a limited market for type theory proprietary things we really meet some international standards and right now there just aren't any and ISO doesn't provide provide those kinds of measures so one thing I want you to do is is go out and help create some international international standards for measuring the actual structural quality of software from the source code so we launched the consortium for IT software quality back in 2010 I I talked with Paul Wilson who has you know hits the sei and Richard so Lee who heads the object Management Group which is an an IT standards organization that works internationally and both of them said they have had some people requesting that especially OMG a lot of the outsourcers had gone through them saying look we're starting to see these measures in our contracts as the equivalent of a service level agreement but the problem is every customer has a different definition of what they mean by security or reliability or maintainability we need standards so we formed the consortium for IT software quality as a special interest group in oMG managed by the OMG to build basically specifications for standards for measuring structural quality of software and we measure it from the detection of weaknesses for instance that security would be you know sequel injection buffer overflows cross-site scripting and things of that nature that you can detect with static analysis and then count those up and then you can normalize that by dividing by lines of code or function points or whatever your favorite size measure is and so we've now got we've produced eight standards so far one of them is now an ISO standard we have a fast track from OMT to is though which is automated function points so it's been you know it's been a vigorous ten years in that world I'm also the lead American lead to the ISO 25,000 Syrian sovereignty and we're revising this offer quality model and trying to improve the the measures so it's we now have some standards in place so you can actually refer to standards for measuring the quality and and the size of software okay so how does the quality and size of software the standards relate to business performance well pretty strongly we've got some some some data in fact a number of data sets that shows good correlation strong core relations between some of these structural quality measures like reliability and and whatnot directly to incidents to operational incidents in the software we also have have shown that reductions in the the most critical structural quality weaknesses can cut maintenance cost substantially because it's just that you know you make the software simpler when you do that so we do have a number of data sets now that are pretty convincing in terms of business performance to the extent at the the problems in the in the Wall Street Journal in new york times now weekly you're reading about major security breaches for major outages these things are now what i call nine digit glitches they're nine digits eight bits or bytes no dollars or euros these things are stretching into the hundred million dollar 100 million euro range and in some cases way exceeding that so and you don't know which weakness is going to provide one of those glitches it's usually some interaction between several of the the weaknesses that that caused the big outages that are so hard to recover from so it does have a huge impact plus the less maintainable the code the less agile the business you just can't make changes fast enough to the code if it's a coochie miss and that means your competitors can be more agile in the market than you can because they can get there faster with the changes to what's in their systems so um as you've gone through this what have you seen is the one of the two biggest myths or misconceptions about measurements and business performance over the years well one of the biggest misconceptions that some people say security is different from quality that there's software security that rubbish the data's absolutely blows that myth to shreds the the the biggest fault what we would call quality problems also happen typically to be the biggest security problems that things that would crash the system also open holes for hackers to exploit though there's just lots of data now that show extremely high correlations between measures of the security to software and measures of the quality of the software that like reliability and maintainability so that's that sort of one myth that the date is pretty much shattered that that that quality cost too much yeah well no when you have a hundred million dollar outage you know look like some of the airlines have had our one hundred million dollar security breach like you saw at Target or Equifax or some of the others you know you don't know which weakness is going to cause that so that the focus has got to be on certainly reducing the riskiest the riskiest weaknesses it within your most critical business transactions in the software sure so what's the biggest challenge now that's people face it seems like they don't react until the bad things happen yeah it's getting executives to understand I mean they really don't understand software and and I'm not always sure that the senior executives in the software side you'll see I always be piece of engineering or whatnot are doing a great job of educating them I mean they they they're not showing them what the level of risk is the people that are now excited about some of these measures actually happen to be the corporate audit group they're a lot of pressure you know they that by law they have to provide a list of the major weaknesses to the business as part of the statements they make to the SEC and the public and as you digitize more of the business more the risk is now in the software that goes for product companies as much as as companies dealing with IT I mean you look at the the impact of the software glitches on the 737 max eight you know the stock price has dropped and you know you look at the risk of security breaches now these just things like that they've got to be able to quantify that and so these measures that we're providing actually provide a level of quantification based on the you know the amount of weakness in the state can be detected in the software in these various areas like security and the liability so that's a big issue they the business still barely understands that they haven't quantified the impact of poor software on the business because they need to state you know what does it cost if we have an hour down time in our reservation system or in our our website where our customers will buy our products you know what's that cost and the minute you start putting numbers on this they start understanding the enormous risk as more of the businesses digitized and it's really developing that quantification that's going to be critical yeah now even on a smaller scale and then one time I left was going on a trip out of the Austin Airport here and the whole Airport system went down they had to go back to you old school paper check people and they couldn't they couldn't do the standing of your boarding pass yeah I mean Delta estimated I think their last huge outage that it ran over a couple of days you know trying to get people back on airplanes with about a hundred and forty million dollars so these things are when they happen or not cheap sure yeah that's the whole thing is trying to get you know you get you quantify it you finally get somebody's attention but it's like otherwise they're in denial and they got a problem well the other gamblers in some cases even if they can't quantify it they'll say you know it's more important to focus on getting this new functionality out then then fixing you know all this you know what we'll call technical debt all these weaknesses and and they ultimately pay a price I mean we need executive policies that are enforced not only by tea or engineering but also by the business for spending some time reducing the technical the critical technical debt the stuff that can crash the system or allow a hacker to break in okay so could you please share a story of how you helped a company to overcome all these like obstacles and succeed well obviously two different worlds I've been in one was for a long time the process improvement world and now I'm in more the product improvement world you look at what happened in Six Sigma in Six Sigma was the rage in the business back in the 1990s and there had to be a certain level of benefit in order to run a Six Sigma project and eventually you ran out of runway just couldn't formulate a new project with enough benefit and so they turned through what they call design for Six Sigma which is our focus now needs to be on the product and how to design it so it's less likely either have we this is so it's easy to manufacture etc spectra that's that's what's slowly happening now in software there still needs to be a focus on process but we're realizing that the product itself is a major source of risk and weakness and cost and so you know that's now being added and focus on architecture to focus on reducing the the severe weaknesses inside the software itself especially those that are security related to the security emphasis now is is extraordinary so the biggest problem we have is our one of the big problems is that a lot of folks inside the business separate quality from development so in essence there we do development oh yeah we got to have these guys go measure you know whatever they're measuring process or quality or whatever that from that measurement is somehow separate from development activities which is absolute rubbish and engineering you don't do engineering without measurements just it's just but make any sense and we need to get that sense in software that if if you're going to do software engineering measurement is a part of what you do as you develop the product I want to continuously measure whether we're building the quality in or whether we're going to kill ourselves in testing trying to find all the screw-ups that we made trying to build this thing too fast and so that's that's a huge issue is getting people to understand measurement is the essence of engineering it's integral to the engineering and development process so that's one getting getting executives to use the measures is another challenge you know getting them to integrate that into their process of how they manage of what they expect to see how they're going to use that and their quarterly reviews how it's going to be using that you know below them and weekly reviews and that sort of thing how it's gonna be used in the middle of Sprint's to determine you know are we releasing simply shouldn't really release because there's thing fixed that are high risk so getting this you're getting this integrated into both management and engineering process is fairly critical finally an interesting problem we find with an a lot of companies you know if you're gonna measure the product the entire IT product the entire application system you've got to be able to find all your source code can't believe how many people don't know where their source code fits don't deal I got big chunks of it but then you'll say well hold it this is Hill calling that piece over there where is it oh I don't know it's somebody go find it yeah yeah it's it's just insane watch them try to locate and integrate all of their source code again Wow so yeah you mentioned about cyber security so I remember several years ago hearing you speak and talk about about a credit card breach and what that did freak so can you relate that to oh yeah well that was one Saturday that was it Saturday front Friday anyway one morning apparently I enrolled for this this was like 2010 or so 2011 something like that apparently one morning I enrolled for the spring semester at Baruch College in the center of Manhattan about 30 minutes to an hour later I had to LASIK procedures performed on my eyes in Orlando Florida 30 minutes to an hour after that I bought four thousand dollars worth of art in Kansas City in an hour so after that I bought ten thousand dollars worth of art in Sydney Australia obviously I don't I don't have supersonic flight powers so city city card realizing I wouldn't supersonic cancel my credit card they're obviously they're they're neural nets and detective this is just doesn't make sense even though the way Kurtis travels so they canceled it and apparently they me and the hundred and forty million of my closest friends that had our credit cards breached there was a famous Heartland breach and so they you like it they but I yo they did a great job within an hour they had texted me called me and emailed me saying you know we've detected that this your carts gone we canceled it will arrange for you to get another number so you know the good news is their systems work thing and they use a lot of neural that's and other kinds of detection systems it worked I feel sorry for the art dealers because they're probably they've probably lost the art I mean they're not going to recover that are the LASIK procedures aren't going to be reimbursed but you know here's the crazy thing how can some kid enroll in Baruch College on my credit card the figure he's not gonna get caught the first day of class for the police walk in that's right even bigger question why would brute college and roll somebody that dumb yeah yeah yeah all the kids the kids dead so at any rate yeah that just was a that was and I Henry here's the worst part of it the breach had been going on for a long time I mean these guys are smart enough not to take 140 million records out in one pool they'll do it over a long period of time they'll take a few thousand s to a ten thousand so it doesn't look strange if you're if you're monitoring what's going on the databases they eventually got caught and it ends up that they had broken in through through some weaknesses called sequel injection in essence the software didn't vet what was being input in the user interface and the guy inputs and basically some sequel commands they got down to the databases please send me your credit card I mean probably more complicated than that but but the hell of it is we had known about sequel injection for over a decade when that happened way over a decade why hadn't they remove those weaknesses from software everybody knows about sequel injection everybody knows that so a well-known weakness is the first one hackers hunt for and it yet we still find those kinds of weaknesses in the software itself so a lot of organizations are doing a very poor job of going in and really inspecting their their business critical applications or customer-facing applications for known weaknesses that are easy for hackers to exploit yeah that's amazing and and when you start stacking things together you add additional vulnerabilities and nobody even understands yeah yeah once you get more four languages on multiple layers calling other systems it gets pretty it's gonna be especially scary when we get into the internet of things and they can bring into one system and get over to your pacemaker right yeah I mean that is really scary or where your refrigerator and take your house yeah so thinking about cradle degrees when you born to where you are today what defining moment in your life inspired you to be an advocate for software quality it's a long story my yeah what collagen as a math major got fired up about behavioral science so I really went into doing building mathematical models and the Social and Behavioral Sciences as an undergraduate went to graduate school and ended up spending more time studying statistics and experimental design and measurement theory and these things although my PhD is really an organizational science organizational psychology but I spent two-thirds of my coursework on the computer you know doing data analysis and whatnot learn I learned Fortran without knowing I'd actually learned Fortran but I didn't know if the user interface language for SPSS was Fortran four and I get to space division they hand me a download program from a satellite and I said oh you guys use SPSS they said what are you talking about then Fortran so but at any rate so I got I got my first job at University of Washington of teaching statistics in the department's I can do an organizational research and I because I lived in the computer center analyzed a lot of data the chairman of the department asked me to be on a dissertation committee for a fellow that was doing work on the psychology of programming and the fellows name was Tom love some of you may recognize that name as one of the two guys that created Objective C so which is here was Apple's big language when Steve Jobs came back and took him over but at any rate Tom then Tom hired me into GE space division to come help do statistics he was studying whether you could predict programmer performance with halstead McCabe's Metrix whether things like structured programming actually improve programmer performance things like that so it was behavioral research but in software engineering and and the first time tom offered me the job of turned it down because it was a soft money job and I was tired of soft money on to hard money jobs or what the warehouser is their organizational psychologist and pretty soon they fired my boss and killed my department so pretty soft money so Tom happened just called me back one day and so I went fine let's go give me an airplane ticket so I went to cheese-based division and started doing this really behavioral research on software engineering and just fell in love with it because it took me back to my you know my math background and my background and living on computers and holy cow I actually knew Fortran for and didn't know it but at any rate that that that really was the the launching event of getting me into software and studying this and measuring measuring software and seeing what we could predict with those measures and then from there I was hired into to ITT the back then company to build a database on productivity quality of their far-flung software operations because you know all of telecom back in about 1980 was switching from relays to digital systems and most of these companies really didn't know much about software and they were scared of it so we went in to try to measure the productivity measure the quality help them help them figure out what are better practices when you're trying to measure this intellectual but if the intellectual artifact it has no physical instantiation so I was there for three years ITT ultimately lost out it was clear that several of us they were collecting the data they weren't going to make it at about the same time MCC was starting down in Austin the big fifth-generation research consortium and I've been doing some user interface work for a while especially up in ITT and if I got hired to build user interface lab and try to integrate AI techniques with with with good user interface practice and the down then lesbo Lottie came to run the software technology program he brought me over and this was another critical thing he offered me you know I said which program do you want to run and and I thought if I thought I'd take the interface program to interfaces to to case tools basically but I saw the process that design processes being by far the most interesting so I said let me run that one and we started we started really studying what do people do when they're building large systems you know we did we we videotaped designers doing design trying to design a system we videotape teams going through a design process we went on interviewed large project teams on really really huge humongous projects out in in corporations and that really started blowing the shreds out of the existing theories of how people design software I mean the top-down stuff was was cute but nobody did it that way in fact David Parnas had written a wonderful article called top-down design and why to it the notion was when you're finished you want the documentation to look top-down because it's easier to understand and see what's in the system but that's not the way you're gonna do it that's not the way you're gonna actually build a system and we had videotapes of guys just really going all up and down the levels you know they have a design idea but they have no idea if and work so they drill all the way down to almost the code level to say would this work out if I actually did it this way and then they'll come back up at the top so it was this constant moving up and down the various levels of design to see what was feasible and work so and then we started building tools to try to capture that way of working that would allow people easily to move across the layers of design and communicate effectively across the team Watts Humphrey happened to see what we were doing and so this was a seminal moment he asked me to be on his advisory board as he was trying to formulate the process maturity framework so I go up to Pittsburgh three times a year with a bunch of other folks that he had invited and we'd sit there for several days with lots and just argue like the dickens the bound of this model and what it was and say watch can you just give us a concrete concept of what you're thinking about here and and he would struggle in the week struggling with him and work on it but wasps could see that I actually understood what he was trying to get at and and and started believing that there really was some some sense and in the constructs he'd come up with the wait organized model so he when he decided to retire and step back he asked if I would come take his job and and so that that was you know that whole thing was the seminal to they going to the sei taking once his position as he stepped back at that point all we had was the book and the questionnaire and there really wasn't defined assessment process it was really people would go out with watts and we'll go around the halls with amen and sit in the room Holly conducted these interviews but we had really had to formalize all that so I launched the process the project to formalize the model make it and maybe formalize the end of the CMM formalize the assessment process and and and so over a couple of years we we were able to accomplish that and and so there was a lot going on the process program I had a measurement group reporting to me and Anita Carlton was running that and she did a great job of really producing some very good documents defining measures that for DoD that they could use for size quality effort and schedule and so there was just a lot of things we produced the problem was I couldn't move my family up to Pittsburgh for the whole host of reasons we couldn't even get the mortgage back on the house because that was back with me there was an oil crisis there was a sound crisis and people were basically walking away from their houses if they had to move so so my wife and kids stayed in Austin and and then at this I promise I'll be back in two years and so they they would if I see I was really nice they would send me for conjugal visits back off it's kind of preserve the marriage and so I came home after two years and a couple of colleagues call me Don Oxley and Joyce thoughts who would think who were in the process of forming company and asked if I would like to join in with them and I had started a consulting operation and I had some contracts and so we joined together and form key request metrics which fortunately you later joined is one of our star lead appraisers and so Terra quest grew from the three of us the Oakley we had about 40 people and we're like 10 10 affiliates out doing process improvement work all over the globe so it was exciting and eventually Borland came and offered us you made us an offer we couldn't refuse we sold it to Borel Borland promptly went down the tubes so you know we all left went on our merry way and at that point I spent little time at McAfee helping them get some processes in place to deal with a consent decree they had to comply with and then the cast cast found me they then they wanted to form an advisory board on software quality and measurement and I was recommended to them and we talked and they realized we are we ought to bring this guy on board so I and I looked at and said you know this takes me back to my software measurement days and this fits right my alley I love the opportunity to go and especially since they want to build international standards this is a real chance to have another big impact on the world so I joined and we form cysts I've done a lot of internal work in cash every two years we publish a report called the crash report they'll cast application crashed research on application software health and basically we look at the day that we've collected globally on the structural quality of applications and I think this year we'll have over twenty five twenty five hundred applications from from North America Europe and Asia and and we analyze those will say you know here's where the world stands on security and reliability and maintainability and what-have-you and here's what we see are the major factors and we fact I'll tell you two of the things we've seen number one we've seen that the hybrid development methods are more effective in terms of structural quality than either pure agile or waterfall but it's that hybrid that says we're going to do the short of the waterfall style upfront to make sure now these are large applications are not small they're not small things that we get the architecture right because you really can't react architecture you can kill it but it's hard as hell to refactor it so you start to make you don't have to have the whole thing to find but enough that you know you're going in the right direction and it'll hang together and then you start the rapid sprints that where you're rapidly building and testing building and testing so you're you're making sure you do quality if you do it right you make sure the quality doesn't get out of hand because you're evaluating that at the end of each sprint so and hopefully they've protected some of some portion of their stories in the next in the next sprint for retiring some of the technical debt they detected that's this dangerous so we find that that that method that hybrid method works best second thing and the comm world will like this we found that yeah there really is a correlation between CMM levels and structural quality and the difference is really the difference between level one and levels two and three that level one is significantly lower in structural quality than levels two and three and you know you're not know I don't think anybody in the lead appraiser will be surprised because they see it every assessment where a level one is just out of control and and they just can't keep up with the quality problems and and whereas at level two the one of the first things you do is you get control of commitments and get control of the the baselines and so you you start putting in place the processes and the infrastructure to control quality so you don't throw people into chaos mode and you do did you do do some control over the the product itself we didn't see a big jump in structural qualities you go to level three the big jump is going to level two level three is more about getting an economy of scale by having a standardized process across the organization we don't have and that we didn't have enough data on level four and five to to show what what happens when you go to those levels I know I've seen some data that says it's pretty dramatic but we we didn't have data and have enough level four level fives to evaluate that and what we were doing but both CMA level and design method development method did have impacts okay so then you'd share an experience you had growing up that still influences how you approach business and work today Wow playing quarterback me all my life want to be a quarterback my senior year I got to be one and at that point you have tremendous responsibility you've got to organize you know everybody's looking to you to make the decision call a play make it work and and that's a you know if I think well you're just a jock yeah but you're when you play that position you're also a manager your your your the on-field manager of the team and and that really wanting to do the things to get prepared for that the personal changes I had to make I grew up without a father I came in a up in a divorced home and and that puts a lot of challenges John and I had overcome some of the problems that left in order to be able to have the respect the team and and to be able to move into a position like that so that had a lot of impact you always have always was good at math and there's no question that he'll because then you think logically and you appreciate the elegance of a beautifully designed piece of logic whether it's a proof or a program you know and so those two things I think had a lot of impact on the fact I do appreciate really well-designed well develop software and appreciate a disciplined process and good management to help create an environment where you can do that so what do you see is your next challenge or are you kind of just gonna ride this out well I you know I it's I think trying to help drive the maturity concept beyond software you know what's saw this is a software process thing but with my background in organizational science you know the minute I got involved with that model I said man this is this is more than that this is a one that probably the only unique model of organizational development that had been developed within 25 years of when Watson come up with everybody else says you know you start with his organizational stuff and watch said no that doesn't work we've seen that not work for 50 years you've got to stabilize local work first and that's what level two is that was that was a unique insight me basing it off Phil Crosby's model but Phil started you know films was a continuous model I don't see that working because it's not organizational it focuses on individual processes and not the organizational transformation and the power of the maturity models I saw it was that organizational transformation and watch saw pretty much the same thing up but I realize I can take this concept and apply to a lot of other dimensions and as we were getting to CMM out you know I was games director of the process program I was getting calls from lots of people saying look this is great it works it helps us get our management our configuration control and and all these other things you know in order but we got all these people problems and you didn't say a word about that we need help you know we've got retention problems about compensation problems and they went on the list and so I launched the people CMM and you know I you know and and he was five people that how can the software Institute come up with an HR model but I didn't tell people in my background was actually organizational psychology which is you know that's their field is all of these things so but I basically built the people see me integrating that the full suite of Human Resource human capital management practices into the maturity concept and we had to get it right you know and and wasas victim was you know you've got to stabilize the local management first and so we we got the abstractions right to say the first thing we've got to do is help people that supervise local work be able to manage the the workforce issues in stabilizing that local work staffing you know perform managing performance dealing with training and development needs and in providing the resources people need and all that so we got that we got that abstraction right and built that and you know in NDA got heavily used across most of the major some use in the u.s. some use in Europe Ericsson certainly was adopted it but not as much as as we had hoped but you know in India used it got there a lot of them got up to level four level five and now I think what they're saying is it we're not really worried about compliance to the model but we do go pick the best practices there's a lot of things in that we realise when them improve our ability to retain people and build skills and that sort of thing so so that made me happy now that the the next objective would be to get this out into areas that desperately need it like health care education you know other areas like that we do have a business process maturity model that is a standard with the object management group and it's a true maturity model I'm not fully happy with it because when we built it we had generic practices in it I've been against generic practices and CMMI got formed because I think bureaucracy in excess that blows the model badly blows the model and so we'll probably have a revision of it at some point in the future we may build a meta model that you could then apply to any domain if you get the abstractions right to taking the healthcare education but the first thing you got to realize is what constitutes local work it's not the school it's the classroom it's not the hospital it's the ward and so when you get those abstractions right you can build a maturity model that really has tremendous impact on changing an organization's performance sure so think what about startups and entrepreneurs how can they benefit from this because they're all they're looking at is good ideas and trying to come to market as fast as possible with a minimal Viable Product yep yep well reading that book the Lean Startup is a good place to start and and you know I was I read it and I was amazed at how much he talked about processes yeah you know it's not like just y'all get out there run free and just hack away and build as fast as you can he actually talked about a level of this if you put a crummy product in the market the market is gonna try it and drop it if you build a good product that people like that they see as an advantage then they'll come back for more so now here the biggest problem with startups is that the founder doesn't survive the the next big jump in size and what not most founders are basically run out by the by the venture capitalists or funders or whoever because they simply can't manage the company at the next level of growth I think a role model here you ought to study is Michael Dell Michael started the back of his of his station wagon and now runs one of the largest computer companies in the world and and Michael was smart Michael you he didn't know how to manage at the next level so he got him a board of directors that could guide him and tell him what to do and help him find the people he needed to manage at the next level up and so my advice to young entrepreneurs is what number one discipline is important quality is important for what you put out so that people stay with you but also be aware you may not have had experience if you grow fast at the next level up of management find people like Michael Dell did who can guide you and what you need to do and what kind of people you need to bring in to to manage the growth okay great well thank you so much bill just one last question how can anyone get in contact with you if they want to learn more about what you're doing with castor just you know ask questions like is approval process privet embezzling you can get a hold of the email Curtis C ort is at a c-- m dot org i'll have that email address and be active until a year after I'm dead because you'll think it you're for a cm to realize I haven't paid my dues and shut down the account you can get my telephone eight one seven two two eight two nine nine four but email me first because if I'm overseas you'll wake me in the middle of the night I won't be very happy but and you can eat you can mail snail mail me at P oh one two six zero seven nine Fort Worth Texas seven six one two six - zero zero seven nine but I encourage you Curtis of ACN org happy to get the email and happy to respond and and help you in any way I can well thanks so much for your time today this has been wonderful talking to you I've learned more about you today and I think I've learned those you were too busy doing assessments back then okay well thanks so much for your time you [Music] you