Verify Initial Zip with airSlate SignNow

Verify initial zip

so I'm just a quick survey before I start can I see by show of hands has anyone ever heard of formal verification before okay this is a good start has anybody not heard of it okay well let the record show the number zero here next question has anyone ever used formal verification okay that may be about 40% here okay now next question who thinks their design that has never been formally verified works ooh you're gonna get yourself in trouble without what I do yeah okay we'll say about 12 people there and okay two more has anybody ever read the zip CPU blog hey I've got a little bit more than 50% we'll say maybe 60% okay now I'm gonna be surprised if I get much of a response in this next one has anyone tried the zip CPU hey I got one I am so good okay those are my quick questions here so I want to talk just a real quick what is formal verification I've only got two slides on that looks like you guys have all heard about it my goal was to formally verify the zip CPU so I'll spend a moment I'll say here's what I expected before I start and then I'm gonna talk about a bunch of things you know sort of lessons learned as I went through different types of components about doing this formal verification and finally I have no shame I will tell you the bugs that I found in fixed using formal verification things that none of my test case has ever found hopefully you'll get a little interested by that and like you said I'll be since learned along the way up whoever's got my time if I start running short on time I'll scoot through these lessons learned you can read them in the slides later okay there's three basic types of formal verification properties there's the assumed statement which tells the solver if this isn't true don't even bother looking at it there's the assert statement which says if you can ever make this false you win it's almost like a competition the guy's really out to get you and then the third one is a cover statement where you prove that something can happen okay basically you can sort of state it as given that my assumptions hold prove that my assertions hold sort of the basic idea behind it as one person said when I taught a course on formal verification he said the saw was really quite - bastard isn't he yes he is he is out to get you every way he can and that's just sort of the way to do it the other thing you need to know is there's this past expression which gives you the value of whatever was inside that x1 clock ago okay these are the basics you can do a lot with those basics and that's these are what I used for the zip CPU okay the next thing you need to know is there's two basic well there's a third but I'm not going to cover it two basic ways of using the formal tools the first one is a bounded model check this bounded model check will start at time 0 and it will go many steps to go into the future that's number one the second step type you can do is what's known as K induction where you say if I find any place where all my assumptions hold all my assertions hold that same number of steps find if you can make it fail at the last one okay they're very different types of proof although there is the mathematical mathematician will say you know for a given length there is a length where BM see an induction are equivalent that's as far as I got in the paper before I got lost the big thing to know about the difference between the two VMC you can do really really simple properties if you want to pass induction you've got to have invasive properties you've got to get into you're designing you've got to tell the engine this is what can never happen you can do bounded model checks on black boxes if you want to do the induction you've got to get in there it's a it's more of a white box test an example of the bounded model check of the CPU is the risk 5 formal approach because it's a bounded model check it can really on lots of different types of CPUs the proof I used for the Zipp CPU is very specialized to the Zipp CPU it's an induction test takes a little bit of time to do that BMC it'll find failures the induction will prove the properties for all time all right they both have their purposes I'm just drawing a couple differences between them so before I got started with working on the Zipp CPU I had formally verified all the components I had verified several of the peripherals timers counters interrupt controllers components I verified the prefetch the instruction cache the decoder the ALU I'll discuss some of these along the way you know I've gone through all the simple stuff why because the big thing was intimidating all right let's be honest that's why I was worried about the big one only the top level of the CPU remained initially I was concerned with pipeline bugs like instructions that would vanish as they went through the pipeline because some following instruction passed it I was concerned with instructions that might get duplicated or anything where a register would be written to and it wouldn't get propagated through the CPU this is what I was expecting you'll get to see at the end what I actually found if you're not familiar with the zip CPA the zip CPU basically has um let's see five pipeline stages there's the pre fetch the decode the read operand stage I'll show you that in a moment and then there's this execution stage where there are four different types ALU memory access divided and someday I hope to put a floating-point unit in there I haven't done that yet but I show it on there just because I'm hopeful and then you have the right back at the very end so one of the things I need to verify are to make certain of is of all those execution units only one is ever allowed to write back at any given time it's just the way the CPU structured other CPUs will allow to but here I can set you know here's a set of three different things that can write to the register file either there's they're all zero or one of them is true can't never be two or three true at the same time here's another unique thing to the mega zip CPU once an instruction hits the memory unit conceptually it cannot be rolled back so if you want to do a load you have to make certain that any conditional execution whatever that would turn that load on or off is completed before the load starts this is designed to allow memory operations that have other consequences like reading from transmit Q or something from a hardware let's see operands going into the execution units this is one thing that really gave me a hassle there's a lot of code working through the bypass the execution units to make sure the right stuff goes in so it's got to be matched the register state the wishbone interactions have to follow wishbone properties instructions from the prefetch are constant until accepted this applied into several stages it wasn't just the prefetch but these are just sort of some of the properties I started with just to get myself going into what this would take see I already did that one so real quick on the instruction cache I like to think instruction caches or caches in general are really easy to verify there are three properties you need so to start you just pick an arbitrary address and an arbitrary piece of data I'll show the instruction cache version the data cache isn't much different number one you assume any time you get an acknowledgment and the return to address you're expecting equals this address you assume the data matches number two you assert that if it's in the cache it must match the data number three if you return an instruction here and the instructions address match that address then again you assert so we've got one assumption in two asserts now assert that the data is right this is easier than I thought it would be I wouldn't the cash is hard okay let's be honest if you've built the cash it's not simple to build a cash this verification is easier than building the cash there we go when you get to an instruction cache another one's very powerful is you just cover the valid trace and if you're just starting out you say give me any trace through this instruction cache that gives me a valid return on the cache it's amazing how many bugs you can work back and forth with the formal tools to find a lot of what's going on through there you've got to see a reset in your trace you've got to see a CPU request an instruction it's got to get a cache miss it's got to fill the cache line it's got a all-in-one trace and oh by the way that's just one line of formal verification stuff made that happen I could do a whole lot more this just gets me started real fast with the data cache I did the same methods you have to modify them a little bit because the value might change and I had one bug in simulation I was I felt like I was doing so good I verified that it would return the right value but I never verified that it would return a second value it left the I'm busy flag high after returning the value oh got close but it was only the one bug so the formal verification got me most of the way there and I have instruct a sh bugs there's really just so hard to find so formal methods found in my case most of the bugs the tools can return really quickly we're not talking hours and days for most of the proofs the trace it goes directly to the bug you don't have to go looking through 100 gigabytes of trace file to find a bug actually I'm usually working between one and ten gigabytes when I deal with traces when you're using the formal tool the traces may be 10 kilobytes it might have 20 clock cycles in it that's all you have to examine to figure out what's going wrong when I got to the full zip CPU I was only doing 18 clock cycles was all that it took to prove the whole CPU let's see minimum logic steps I mentioned I could do it in eighteen but from this data cache I can't remember what it was for the data cache but for the data cache I still needed simulation when I was done so it got me most of the way there in the instruction decoder here's my first lesson learned I had a whole lot of if the last instruction was this then the output of the decoder should be that on the next clock cycle so if I'm on the last I here's the instruction I'm looking at then here's what's my output should be the problem I had was the check doesn't apply any time the pipeline is stalled so if the data cache isn't going through this transition because the pipeline is stalled the formal tools aren't checking anything and so this allowed it to get into states that it shouldn't have been in had I just kept that instruction around one more clock cycle and then verified this DC BR against the net clock cycle I wouldn't have had this problem if I do this again that's what I'm gonna do the that's what I just said verify the outputs instead of the transitions that check would apply even if the pipeline is stalled okay the next thing that just got in the way way was debug port I said I'm gonna be so smart I'm gonna put a debug port into my CPU and right here it's if the ALU isn't being used if it isn't right I'll stick my debug port there and it's gonna go right into the reg file and I'm gonna stop things right at that point if I ever go into debug mode what the formal tools showed me that I wasn't expecting is if you've read if something from the register file added an immediate to it and then dumped it right here and so you're waiting for permission to get into the ALU for the debug port to be free it'll write to that register file and invalidate this and I had never put that logic in to invalidate it this is one of those things that I found but I wouldn't have found without the formal tools yet no end of problems yeah I thought it was so simple when I was building it I discovered it wasn't nearly as simple as I thought it should have been the problem was the pipeline my solution was any time you read write something that debug port just reload the pipeline you're slow anyway you don't have to be fast when the debug ports and classist can anyone else agree with this one alright abstraction so this is one of those things I really want to write about on my blog I haven't gotten to it yet but the idea basically is let me replace the multiply with an abstract multiply and all this abstract multiply is going to do is it's going to return a number that the formal tools are going to select it may or may not be the right value for the multiply I know that the multiply works I just want to know that it works inside the CPU so I would replace the multiply was something that I called an abstract multiply which would say that if either the incoming values was 0 the result should be 0 otherwise it should not be 0 the solver picks the result solver picks the input and output values and up inside the CPU the whole thing would fail and so this works real nicely for formal verification it requires you to maintain the signaling improve the CPU logic works made things pretty easy my problem was the top level CPU has worked proved it worked and proven but I didn't formally verify all the modes of the multiply itself I got past the multiplied with the abstraction but yet there was a bug in the multiply that I didn't catch lesson learned create a property file for every interface that defines both sides of the interface when you're proving the CPU use one half of that property file when you're improving the multiply you use another that would apply to the prefetch the decoder the ALU and the memory unit still haven't done that this is what I would have done because I got caught on that interface next piece is aggregation one of the real things formal tools it the basic lessons of formal tools never go over aggregation it's not your first lesson the problem is I've got several components to my CPU and each of these components has assumptions and assertions within it if I just put all of these pieces together these assumptions are going to keep it from finding problems so what you have to do so I proved every component before beginning that that's the first step but then what I did was I swapped the assumptions and the assertions so if it was an assumption before it became an assertion if it was an assertion before it became an assumption so before this would be saying as long as this is true prove that that's true now it's saying prove that those assumptions are actually true in the first place I got caught on a couple of assumptions I've made within for example the prefetch that didn't hold so this gives you a whole new set of CPU properties it turned out I had some assumptions within my memory controller that took a bit of work to prove lesson learned the sub-module assumptions aren't Givens manage those interfaces make sure both sides of the interface maintain the same properties they need to be proven as well so the Zipp CPU was built with a movement with a multipass verification idea so the idea is if you've got a whole set of assertions you start with your assumptions then you prove the assertions on the first step then when you go to the second step I don't need to prove that first set again I can just prove the second set I could have gone on with the third I only went with two for the CPU anything you've proved can be an assumption to prove something more must be done in order you can't assume stage 1 up here until you first proven it via assertions and any logic change will send you back to the beginning so what happened I had two passes I used one just to prove the component assertions I had a bunch of ad-hoc assertions and some pipeline assertions and then I'm the second pass I said let's pick a specific instruction and let's follow it through the reality was that was a waste of time the solver picks the set of instructions in the first place if I had just followed those instructions on the first pass I wouldn't have needed a second pass and then I had to prove most of the logic anyway on the first pass just because the proving the stuff within the sub was that difficult okay what did I find so we have some fun here bus there on an instruction read didn't necessarily halt the CPU so if you try and read something that's not memory the CPU should halt with the bus error it wasn't interrupts might break the compressed instruction words yes the zip CPU has instruction words the debug Reggie bug register if you have an instruction word on the zip CPU it's not two separate instructions it's one word with two instructions between it they go as a unit that gets me out of all kinds of painful logic of trying to handle misaligned instructions everything's always aligned well if you halt mid compressed instruction Perique divides would start before the multiply finish so if you have an output of a multiply that you wanted to stuff into a design divide it might get started break instructions would be I'm glad I did this it was very useful lesson learned before using I did a lot of programs with the Zipp CPU I applied the CPU to many FPGA boards debugging on an FPGA is difficult can I get an amen brother simulation requires lots of gigabytes of traces with formal I was a simulation alone didn't find these bugs even an incomplete proof was valuable what you don't prove will surprise you simulation requires gigabytes of traces versus formal was about 20 to 60 kilobytes still needed simulation when I was done and you can take a simulation symptom and recreate it within the formal tools to see what would cause this to ever happen if I had to do it over I'd start with formal formal seems to have such a benefit when you're just building something in the first place it's an amazing benefit to get something right the first time I would do formal before I ever touch simulation I wouldn't do it the other way around and I definitely do it before a code bloat code bloat just adds so many options that then need to be proven it makes the proof all that much harder to do all right did I finish some time I'll take some questions you would mentioned if you would change the code you'd have to start back from the beginning on your formal pass yeah so is there a way to set up your formal verifications so that it or it you can keep both sets so that if you're running your formal verification you go in and make a change your tests are just gonna run again or you need to go in and if and physically modify the code so the way I've got the make file set up is if certain code pieces change the that proof needs to be redone based on those code changes if you try and do it without redoing the whole proof you will get something other than formal verification that's what formal verification does it goes through everything it proves everything I had problems where I had proved the second pass without the first and then I found a bug in the first pass and it's like why did I waste the time on the second pass you get the idea second question I'm sorry Luke I was wondering did you use or have you used fault injection or any other techniques to sort of prove the completeness of your assertions and assumptions I have not that will be another experience for me when I do perhaps I'll come back and discuss it you have a question Jeremy or just a comment I like your um quotes there but I would commend to you for our formal verification Acts chapter 2 verse 12 thank you very much someone can go look about I will be doing that as soon as I get down here anybody else sure please which tools were you using I mean you might have covered it in the beginning but symbiosis simius this is all done with symbiosis yeah well the the what do you call it the engine I used was the ISIS engine sometimes called yikes depending on whether or not you want a hard C or a soft C but yeah this was all open source I didn't pay a dime for any of these tools oh that does it does it emulate proprietary solutions or was this a brand-new solution so I have a kind of it uses the formal verification constructs that are part of the vera log or actually the system vera log standard okay so the symbiosis only uses the immediate verification constructs there's another version that uses the full range another version of symbiote stirring the version I was using just used the immediate constants what got you into exploring all of this the very good question what got me into exploring this was I was asked to try it out and so in standard human fashion I said I know everything I'm doing is right let me just prove this tool is messed up but just be nice to the people who asked me to try out this tool and so I gave it you know my FIFO yeah what can be wrong with a FIFO a Fife was just a simple construct and I started writing some assertions and I discovered my FIFO didn't work like I thought it did because the formal tools had the ability to imagine situations that were beyond my creativity to put in the test set and so okay well it found a bug in my Fife oh let me try it in something else I found another bug okay let me try it in something else and pretty soon I feel like my credibility is on the line because I keep finding latent bugs in my own code and so I started working through all of my code trying to find all the bugs and then well can you do the I mean this really that the zip CPU proving a whole CPU core sounded to me like big cojones you know can you actually do the you know swallow the whale and yes you can here's some lessons I learned in the process I could do it better in next time if I were starting over but I did get to the end where it worked well well done thank you so do we have a last question how long dusty um to prove everything right now may take a half an hour to an hour the formal process of putting everything in was maybe two to four weeks it would have been a lot faster if I had started with the formal when I first started the core one of the problems is as I've moved this court hardware after hardware after hardware I've kept having to trim it or expand it and so it's got all kinds of features within it you can use this feature of that feature pick what fits on your and I had to prove it for all the different features symbiosis if you're not familiar with it has the ability to say I want to prove it where the parameters are this this and this and so if your core is parameterised you can prove all of the assertions given these parameters are true so okay great well thank you very much thank you [Applause]