Create and Manage Your Dropbox Invoice for Supervision Seamlessly

Dropbox invoice for Supervision

let me just start off by asking who uses Dropbox a lot of people okay what about Google Drive oh yeah almost everybody uh one drive a few of us so we're in good company I got some usage figures for these three services uh which are fairly recent so Dropbox is claiming about 400 million users probably more by now uh Google Drive 240 million one drive quar of a billion so overall there's almost a billion people who are trusting these services to uh store their data from them so they're really critical infrastructure and this is just the beginning there's also any number of open- source file synchronizers you can use so there's a lot of software of this sort so it's very important we trust these services with our our important data and so I want to ask are they trustworthy really to put it another way what do they even do well you might think that's obvious they synchronize files but what does that mean and I'm also interested in the question how can we test them why that last question well some of you may know that nowadays I love testing but I don't like writing test cases by hand no no no that's far too much like hard work for me I like to generate test cases and that is my joy and pleasure nowadays why because as many of you will know for many years now I've been working with a random testing tool called quick check this dates back to 1999 when K class and I uh came up with the first version for hll and 10 years ago now I founded a company marketing edang version which I've been spending more and more time with since then and over the last 10 years we've made lots and lots of extensions and tested lots and lots of interesting software it's been great fun so the purpose of this is to generate tests let me just summarize why might you want to generate tests instead of writing test cases by hand well for a start because you will generate a much wider variety of tests than you would ever think of writing by hand because test cases are so cheap you can have a much wider variety and you'll find generated tests will will do things that you know only a nutcase would think of doing but they can provoke faults in software so because you have this wide variety of tests you end up with more confidence that your software is behaving as it should and that's good and on top of it all you do less work because you don't have to construct each test one by one so it's really a no-brainer is isn't it you can see why I like it so much I'm just going to show you a very brief example of how we generate tests with quick check um I usually talk about this in a lot more detail but we're just going to have a quick introduction for those of you who haven't seen it so imagine that we want to test a q so we might have an API like this where we can create a que with a given size we can put a value into the que I'm just going to put in integ and we can remove and return the first element of the que if I test this code with quick check then I might end up with a failing test that looks like this okay so in fact I've made a bad implementation of cues I tested it and quick check generated this test for me so let's have a look at it what's happening well we start off by creating a que of size one that returns some kind of pointer then I put put a one into it and then I call get I get the one I put minus one into it I call get I get minus one and I put four things and this last call of get returned the wrong answer so a post condition failed I got minus one and I was expecting zero so this is the kind of random generated test that is likely to fail and if you look at it I think we can agree that it's long and boring and this is the problem with simply randomly generating tests that when they fail they're very rarely Salient tests they're going to contain something that makes the code fail but also a lot of irrelevant junk and that is a real problem if you try and debug test cases like this so the other thing that quick check does is once it's found a failing test it tries to shrink it and it shrinks it down to a test that is so small that if you remove anything from it the test will pass and in this case we'll end up with this test case here and this is only four steps long and if we look at it we can see that first of all I made a q of size one and then I put two things into it no wonder it didn't work and you can see that the result of get here the wrong result of one that's actually the second thing I put so since we know that the Q is supposed to have space for one element we can guess that maybe that one overwrote the zero that I previously put in and that's why the test has failed that's actually what happened uh this implementation by the way is a circular buffer which I programmed in C I expect all of us have implemented a circular buffer in C once probably a long long long time ago but I hope you remember how painful it was and how difficult it was to get it right so I I use this as an example quite often so when you see this this test that fails now it's quite obvious what's wrong but there's still two ways you might respond to that you might say well there's a bug in the code when you try to put that second element in it shouldn't just happily overwrite what's already there you should get some kind of exception so that's one reasonable answer and if that's my interpretation then I'll fix the code and I'll extend the tests and everything will work better but there's another alternative which is you might say what were you thinking putting two things into a que you said should only have room for one so maybe we expect that cents of this code will take responsibility for not overfilling it and in that case perhaps we don't want to generate this test and in that case the way I would respond to this problem would be to change the test generator so that we don't see this problem anymore so those are the two ways that we can respond to the report I want to emphasize when when Quick Check gives an example like this what it shows is there's an inconsistency between the test and the code it doesn't say where the fault is it says there an inconsistency and that that's generally true so the point here was that I didn't have to invent the test case I showed you I just constructed my Quick Check tests ram my code and quick check says consider this example and that's that's what Quick Check does for us just summarize we take an API that we want to test Quick Check generates random sequences of operations we keep doing that until one fails when a test fails it's normally full of irrelevant nonsense as well well as the few operations that cause the failure so the final step is to figure out what those operations are and get this minimal example which is what we report to the user and we think of this shrinking process as it like extracting the signal from the noise of random tests random tests contain a lot of junk that is their purpose that's why they're good at provoking faults but they're not good for debugging so extracting the signal from it figuring out why why does the system fail that's key to making the technique useful so what I've kind of glossed over here is how we can tell whether or not a test of the queue passed and when you use something like quick check a test case generator that's more difficult than it is when you write unit tests because we have to be able to tell whether or not an arbitrary test passed so when you write a unit test you know exactly what the test is going to do and you can predict what the result should be but when you generate a test you don't know a priori what the results should be so how can we decide with arbory tests pass or fail well the approach we use is to build a model so this is supposed to represent a real system and this is a a simple model of the system and it looks as though the system isn't very like the model in this case but never mind those were the pictures I could find so when we generate a test we're going to take the real system through a sequence of State Transitions and we don't know what's happening in there anything could be going on generally speaking we can't observe what's happening inside the system under test but we can figure out what states the model should go through so that's how we specify this Behavior we model each state transition and then we write post conditions that just look at the output from the real system and compare it to the state that the model is in and decide whether or not they're consistent if we take the Q I told you it it's a circular buffer and C there's a test case that I showed you earlier how can we Model A Q well it's dead easy let's just model it by the list of elements that should be in it so when we create a new queue it contains an empty list of elements when we put an elements in we append that element to the list when we call get we take the tail of the list so the model is a simple functional data structure and the state transitions are simple functional programs very very simple and then we just write post conditions that compare the actual result to the model so the interesting one is for get at the bottom it checks that the result is zero why zero because that was the head of the list in the model before we started so we use Simple functional programs to specify the behavior of much more complex systems okay so that's all very well I can't see the clock up there that's okay so that's all very well and you can see how I can test a circular buffer but what about Dropbox how can I test something like that in this way well let me start off by telling you how we set up our tests basically all the tests I'm going to show you we ran on a laptop um but in order to test different instances we created a number of uh virtual box VMS all running on the same laptop they all running a to in fact so on each of those VMS we ran a Dropbox instance and those instances are of course connected to the network so they're talking to the Dropbox service behind the scenes and then we need to to be able to run tests that would do things on each of those VMS so how did we do that we are an airl in the host and in the guest operating systems and we used air langang distribution to make it simple as P to perform operations on each of those VMS really nice application of AirLand distribution so what did our tests do well basically we read and wrote files in the VMS and then we watched what happened so what we were trying to do here was create a very simple model of what a file synchronizer does we wanted a model that would be simple so that a user can understand what's going on without needing to know about implementation detail and we also believed that um all of these Services they ought to behave very similarly so we were hoping to find a model that would be generic and that we could apply to Dropbox Google Drive one drive owncloud you name it at least with very small variations okay so let's think about it then how might we model what a file synchronizer is doing well we're going to have to model the state of the files so we might say um how about modeling just the content of each file well that can't work so for example this might happen now I'm going to show a lot of diagrams like this let me explain what they mean each Black Arrow represents what's happening on one of those VMS and I'm going to write the operations in the test beside the arrows in order so in this in this scenario first of all we've written a to a file on the first VM and then we read the file on the second one and we get back missing so this red missing I'm going to write results in red missing means the file wasn't there and these two operations are performed in order so you'll notice I'm not giving the file name that we're reading and writing actually all of these tests we only needed one file for but um so just ignore the name we're looking at the file contents so of course this can happen if Dropbox hasn't had time to synchronize the file from the first to the second client then we expect that when we read on the second the file will still be missing so it's not a failed test but if I try to model the state of the system just by saying well a file has a contents then after the first write the contents must be a and the second read would be wrong so that model is too simple okay how can I enrich the model well maybe I could say let's say that the file has a contents on each node okay that that makes sense uh then that would explain the behavior in this first test but here's something else that can happen maybe when we read the file on the second load we'll actually see a so if our model says well the file has independent contents on each node then this wouldn't be consistent with the model but of course this is also perfectly okay expected behavior and what's happened here is that behind the scenes in the background Dropbox has copied the file from the first client to the second so this is the problem with testing something like Dropbox usually when you test a system your test is controlling everything but in this case we can't control what the Dropbox client is doing in the background we've got to test a non-deterministic system and we've got to accept test results that may differ depending on what the Dropbox client happened to do now of course a lot of this behavior is timing dependent so maybe in the first case I waited half a second between those calls and maybe in the second case I waited a second the tests that we actually generated contain random times so that is important for provoking some of this Behavior but I'm not going to show you the times in each case because those might vary from case to Case and in any case you can't predict what the Dropbox client will do just by how long you've waited uh one of the things you quickly discover when you try and run tests this kind of service is that random tests that read and write files a lot of the time they look very like a a denial of service attack and what do you know Dropbox has a kind of exponential backoff so we have to slow down the tests enough that um we're able to run them without encountering that so so we have to accept um different Behavior depending on on unknown events in the background and to do that we've developed an approach which is new for quick check at least every operation in the test case we're going to regard as an observation and a state transition so for example if I read the file on a particular node and I get a then I've observed that the file contained a on that node if I write a file and I write B to it then that is going to make a state transition in my model that says the file now contains B on this node and then when we run tests we're going to start off with an initial model State and then we're going to say well what are all the possible background actions that might happen let's look at all of the possible States we might get to by Dropbox doing stuff in the background again this is at the model level then when we make an observation we're going to say oh that's only consistent with these possible States so we can discard the others and now we can take each possible model state that we might be in make the state transition and then do the same thing again construct all of the possible States we can reach with background actions and when we run a sequence of actions we just do that repeatedly so we start in a given model State we look at all the model States we can reach with background actions we make we select the ones that match the observation we've just made we make State transitions background actions select the ones that match the observation and so on and so so forth and by the time we get to the end of a test we're likely to have a path that explains how we could have reached this model state in terms of the background actions that Dropbox could have done along the way there could be more than one explanation and that's okay but if there's no possible explanation for the test results we see that means Dropbox must have failed or it means our model is not consistent with Dropbox so what background operations can Dropbox do well I suggested earlier that um it might copy a file from one client to another but if you look at this scenario which is possible then you can see that since the second client saw an a when it read then we must have copied the file after the first write so that explains that read but it doesn't explain this one and in fact the second read can't be explained in this way because we saw a b so if we had copied the file at that point where that gray Arrow has appeared then we would have seen b at the second right but we would have seen b at the first read also so we can't have copied it there and we can't have copied it here because then the second read would have seen C so we can see from this scenario that Dropbox is not copying right in the model we cannot model Dropbox as copying a file directly from one client to another in an atomic action so we have to extend the model how do we do that well we introduced a server buffering in other words so once we do that then we can say well we explain this by saying that the copies to and from the server happen in this order now notice we can't observe OB erve the server we're not even saying that there should must be a server it's very unlikely that all these clients are connected to the same server at Dropbox there's probably load balancing going on what's more our VMS are all on the same local network and we know that Dropbox optimizes copying between nodes on the same network so this isn't what's really happening but what we're saying is we're going to try to understand what Dropbox does as though there were a server to which files were uploaded and download it and we've largely been able to do so so this is a way of thinking about what Dropbox does so there's one more thing that I have to talk about before showing you the actual model and that is conflicts so um this is one possible scenario that we write a to a file on the first client then we write B on the second but when we read the file we get an A okay so that is all right and the explanation is that the a was uploaded and then it was downloaded after we wrote B on the second client but if you look at that it looks a little worrying doesn't it because we saved B in the file and then it seems to have disappeared but actually it doesn't disappear Dropbox puts B into a conflict file and we'll all have seen these things you know files with names like John's conflicted copy December the 25th or whatever so Dropbox tries very hard not to lose data and so one of the things that we want to test is that the right conflict files get created but this is not the only thing that can happen it's also possible that you might read a b there okay so we write a we write B and then we read B now the funny thing is that when that happens you might or might not get a in a conflict file and to tell whether he will or not we really need to know what was the value that that second right overwrote so what we did was we actually when we performed a right we also observed which value we were replacing and if we do that then if the B replaced the a that means that this client had already seen the A and so it's not really lost in that case a will not appear in a conflict file but if when we wrote the B the file didn't exist it was missing then that means that a has apparently disappeared and a will appear in a conflict file so by observing what we overr write we can tell more about when a conflict file should or shouldn't be created these conflict files were a pain in the neck because it's very important that they're created otherwise Dropbox may be losing data but they don't Dropbox don't create them immediately we thought first of all surely a client must know when it's creating a conflict it can just create the conflict files we can just look immediately and see if they're there but no uh the clients can't tell actually so conflict files are created on the server and then they trickle down to each file which means that we have to wait for a stable State before we can check for them okay how do we do that well we added another operation to our tests that we call stabilize and that means wait until all the conflict files have been downloaded and stabilize observes the final value in the file so all the nodes should converge to having the same value in the file eventually and the set of conflict files that get created but the problem is knowing how long should we wait and this is the difficulty in testing systems that do stuff in the background well obviously we should wait until all the nodes agree on the file contents and the conflict files but that's not enough sometimes the nodes have the same data but Dropbox is still working away aha well you can look at the little Dropbox icon and see whether it's working so so we should wait until the demon on each node claims to be idle so we do that but you know what that's not enough because sometimes all the clients are happy but the server is working away so what we actually did was we wait until we see what we expect so you know if there's no point reporting a test failure early so if if we don't see the right thing we just wait a bit longer but not too long because when Dropbox fails if it should fail it's very likely to be by one of these things never happening so you don't want your test to fall into an infinite loop at that point so you can see there are ad hoc choices to be made here 30 seconds so what does our model finally look like well the final one we constructed has a global value that's the value on the server and a global conflict set and for each node we track what the local value is whether the node is fresh or stale a node is stale if it needs to download from the server and whether a node is clean or dirty a node is dirty if it needs to upload to the server so that's our model and then we can just explain what each operation does in terms of the model when we read a value well that observes that the local value has to be V of course so remember as we're running tests we've got all of these possible model States and now when we do a read we can throw away all the ones that don't have V on that node and what's the St the state transition for a read nothing what about writing a new value over an old one well in the same way as read it observes that the local value is old and that lets us throw away a bunch of States but then it also makes the local value into the new one and it marks the node dirty it now needs to upload that value to the server what about stabilize well stabilize observes that the global value is V the conflict set is C and all the nodes are fresh and clean you might wonder how can it possibly observe that well remember stabilizer supposed to wait until that's true so by the time we process stabilize you know as we're analyzing the results of a test we know that by the time stabilized res returns all the node should be fresh and clean and we can throw away any possible model states that don't have that property and then stabilize doesn't actually do anything um to the model so those are the three operations that appear in our tests but there's also the operations that happen in the background that we're conjecturing so download is one of those and that when we download to a node that observes that the node is stale it needs a download and clean it doesn't have a value of its own that we'll be throwing away and then the state transition is to make the local value into the global one and mark the node fresh the really interesting operation is uploading from a node so you can only upload from Dirty nodes and as a result it makes the node clean but one of two things can happen if we upload from a fresh node then the value that we upload becomes the global value and all the other nodes become stale so notice that the other nodes the real clients don't know that there's stale this is in the model not something that the actual system can know but if we upload from a stale node it goes into the conflict set so that is how conflicts get created in our model and uh you can see that we're adopting here in our model a principle that the first upload wins if you write the file on two nodes they'll both be dirty the first one will upload its value that becomes the global value and the other one becomes stale and then when that one does its upload then uh its value is added to the conflict set so we thought that seems to correspond to what Dropbox is doing let's run some tests and find out whether it really does and so of course now we expected that all our tests would pass and the model would be consistent with Dropbox but no here's the first counter example that quick check showed us okay so here we're doing wres on two different nodes to the same file and you can see that this second right saw a non-existent file when we performed it so we had not yet updated the second client with the value written on the first and that means that second right is in conflict with the first so we conjecture uploads and downloads like this and now what's going to happen when we stabilize well we've got a value and a conflict so what we expect of course is that we'll stabilize to a file containing one value and a conflict set containing the other but what we found was that the conflict set was empty there were no conflict files but we can see a conflict here what's going on any any clue yeah the contents are the same so Dropbox is applying a new principle not captured in our model yet that says a value does not conflict with itself have we found a bug in Dropbox do you think no this ISS very reasonable Behavior so what we had to do then was fix our model and the fix was just to change upload so that when we upload a value if it's the same as the value already on the server we don't do anything else so that seemed reasonable so then we thought okay now the model must match Dropbox let's run tests oh no another inconsistency okay now here we've got the server that's in red that we can't observe and the three clients in Black because I've got more lines on the slide I'm going to abbreviate the notation a bit so this represents writing a to the file and the Red Dot represents missing okay so we write a to the file we read it on the second client and that means when we explain this there must have been an upload and a download so the server has a as its value now we write missing to the file wait a minute how can you write missing to a file you can delete it so this is how we model deletion we just say okay we'll write missing and then we read that on the other client so now there must have been an upload and a download now the file has been deleted on the server right so its value is missing and now we write another value somewhere else and notice this WR has overwritten a so this client wasn't completely up to date it hadn't seen the deletion and this right is in conflict with the deletion we can agree on that well when we do a read on this other node we got the B so you know the B was uploaded we expected that but we expected it to go into a conflict file so we expected our model said B should be in a conflict file and the file itself should have been deleted but that's not what Dropbox did have we found a bug and Dropbox well they've adopted a principle that missing loses every conflict and you know what it's hard to blame them for that so we had to extend the model again so we just said okay the local value is just going to become the global value if it's uploaded from a fresh node or the global value is missing so missing loses every conflict and now things are much better so you notice that so far what we've been doing is we've been running tests getting these minimal counter examples but then we've been fitting the model to the implementation so you might ask is this ever going to stop maybe all we're going to do is build a model that precisely characterizes any bugs that might be in Dropbox well so why are we doing this because actually Dropbox have thought a lot harder about what a synchronizer should do than we have so just because we call something a model it doesn't make it right of course we have to debug our model and up to this point well whenever we see an inconsistency we have to ask ourselves is this the intended Behavior or is it reasonable intended behavior and so far I think Dropbox have made good choices but at a certain point the counter examples that quick check finds get weirder and weirder so um let me show you a few of the surprises we got Dropbox can delete a newly created file did you know that if you create a file you write a to it and then you delete it and then you do it you write B to it now that's in conflict with a deletion okay then when we write C on the other one and read it it's gone look at that the last thing I did on each client was write something to the file but when I read it it's not there so it turns out that this is because uh the C is I think losing the conflict so B is going to win and the Dropbox client is actually in the middle of replacing the file but there is a short period during which it's not there at all and uh the Dropbox team thought nobody would ever notice but we did so this isn't so bad because if you wait the file will come back but you it's good to be aware that when more than one client is manipulating your Dropbox some files might just disappear briefly something else Dropbox can do it can recreate deleted files see here's one we found where one client writes a to the file then it deletes it and then it reads it and it's come back now of course this is timing dependent it doesn't always happen but it can happen and the amazing thing here is that there's only one client doing anything okay so there is another client running so Dropbox is probably working in the background but only one client is manipulating the file and and it's reappeared a zombie file actually it doesn't you can do the read and the deletion on different nodes that also works but in this case no matter how long you wait that file will not disappear again so you may find that you delete files and then they just reappear and they don't go away so that seems like a bug but the worst one is the Dropbox can lose data completely so suppose one client creates the file rights a to it another client overwrites that we just read it make sure that that right worked B is in the file on this node sometimes it never stabilizes at all so you'd expect that eventually both clients will have the same contents but they don't what happens is that the left client retains the B the right client retains the A and in fact the left client is dirty but but it behaves as though it's clean which means that if you the other client then rewrites the file it's never seen the B remember then when you stabilize the b gets thrown away altogether now to make this happen you have to do that second right precisely the right moment so our tests make sure to do that um but nevertheless it's surprising Behavior so that that was the third surprise I had to show you so let me just summarize what did we do well we found a new way of doing Quick Check testing of non-deterministic systems by searching for explanations keeping track of a a model with some hidden actions and constructing lists of every possible State you might wonder how many possible states do we have to keep track of well we actually gave up if it was more than a thousand and that rarely happened but it was often a few hundred that sounds expensive but don't worry there's plenty of time to update a few hundred model States while the Dropbox client is synchronizing files so this wasn't a problem um we used the minimal failing tests that quick check showed us to refine our model in the first place and again this is something we commonly have to do until our model matched the intended behavior of Dropbox and then further minimal failing tests revealed unintended system Behavior like the those last three examples so I say unintended uh we have talked to the Dropbox synchronization team about these and um they they say that uh they've been able to reproduce the buggy behaviors they point out that you have to work quite hard to make them happen which I would agree with but nevertheless losing data is not something that they consider acceptable so they're developing fixes for the bugs that we've found what else can we learn from this well you know I started by saying what what does Dropbox do it synchronizes files how hard could it be it's a lot more subtle than you might think and really we've only started working on this um The Next Step would be to add directories imagine what happens if you create a directory on one client and a file with the same name somewhere else what should happen What happens if you have an empty directory and one client puts a file in it while another one deletes it there's all kinds of conflicts where the behavior is not obvious and where more work would be needed to figure out what it should be and whether they do it correctly and of course Dropbox is only one file synchronizer there are many more if you'd like to learn more about this and see more details we've written a paper about it uh if you Google you'll find there's a draft online uh it's coming out in April at the International Conference on software testing and here are my co authors the second guy there is Benjamin Pierce from University of Pennsylvania uh who was uh an important contributor the reason why we worked on this problem at all good friend and good good collaborator and uh of course I have to get him the plug if you've got a tough testing problem come to us thank you [Applause]