Print Initial Conclusion with airSlate SignNow

Print initial conclusion

the two things actually have some overlap so I talked about sensitive II analysis last time we talked about a variety of types of sensitivity analysis the very in what is being married structural parameter variables how they're being buried one way versus multi-way sensitive be analysis I talked a bit about model uncertainty structural uncertainty and we talked about varying parameters we then started looking into in any logic um information on varying parameter values on and we saw that there's a parameter variation experiment in any object and actually you'll find I should really send you the reference that you can build up it's a little bit of a mystery when you open the this model up perhaps how they created it I have a separate video of me building off the model like this from scratch um with a parameter variation experiment suffice it is H is one of several experiment types you can add to your model and what distinguishes this type of experiment is that it provides a way of varying your assumptions well okay that's a level comment it is one of a class of experiments will see another one later today where instead of running your model on a single realization a single scenario three specific parameter values instead of that paradigm we are instead going to go to a situation where a given experiment will run your mall several times over now this has implications it has implications for the plumbing and it's been some conceptual but what I'm saying is that with these sort of parameter variation experiments these these experiments are more than the type of experiment we've seen before and played with what's going to happen is this experiment will create an instance of Maine it will run that instance of Maine with a specific set of parameters it will then extract information as needed from that instance of me if any needed information needs to be preserved to be displayed for example on a 2d histogram here it will extract it you'll suck it out of the main object and then the main object will disappear it will despit oh go away and then it will run it again perhaps with the difference I friend and then again that's the difference of names each time it will extract the requisite information it needs to summarize the results across many men particular problems of your off now in some cases it might actually just run your model the same set of friend why would you do that why would you run while not marrying the parameters at all what would make you to do that exactly we saw some of that last time some of this be stochastic and so to speak casta significant shotta when it comes to the documents they can lead to variability in the dynamics although with significant regularities around some trajectory so sometimes we may do it without varying parameters last time we saw that we could actually very parameters within anyone and that will allow your model to be run for different sets of parameters different assumptions as it were running all once with these set of assumptions time with be set another time to be set and we'll keep on running them as you a played out and then a logic provides the comedian waited doing this by specifying whether we want to pretty simple provide ring ranges for values which are systematically examined within some grid in this case say from 525 for the step of five so like that um and we can do that for one or more parameters in which case would examine a grip maybe the 1d grid 2d grid 3d grid horny grid and we start to suffer from the person situation where the number of points in degree rise geometrically with the number of different parameters so we go from suppose each of these has five different options we go from five with a single parameter x squared to provide 225 report before no and and this rise is very rapidly emptied um so you know often we'll we'll set up several variables to be examined in practice this might be with respect to one we saw it and then we can do multi-way sensitivity analyses where we very say two and we'll get sibility will introduce new sources of variability the challenge is the combinatorial explosion the challenges that were were imposing computational burdens that rise in a way that quickly becomes infeasible for exhaustively exam so if we start going up from N equals 45 go up to N equals panic n equals 20 n equals 30 the number of combinations become rapidly infeasible now I talked a little bit yes last time about the fact that often there are several ways to reduce this burden okay and I've taught him talk about three of them here I introduced one at a very superficial level of the issue of dimensional analysis and this ladies and gentlemen is all about what is tomorrow it's about understanding and even taking advantage this dimensional truck that's here specifically we take advantage of the dimensional structure if the quantities we're dealing with to recognize that we can actually do this parameter variation experiment with pure home so for example if we have a rate of infection spread in our model that's measured in you know how fast it spreads is measured in terms the amount of time it takes to reach some outcome versus the spread of an intervention for example also measure in terms of time maybe it's the ratio of the tube that's important and it turns out the dimensions of these quantities will be key in going through an exercise of deriving what things really matter in terms of the models performance I don't have time to go into this today I didn't have time else last time or today but we're going to hold lectures and within the remaining time so I'm going a bit late on this but just recognize that often what not sure is really what's really governing the dynamics of the law is combinations of friends that's not to say one of those parameters doesn't make a difference in dollars a given one of the parameters it just doesn't l length of time maybe between events or length of time as a trial through this alpha is how quickly it occurred but make a difference sure but what really governs it at a deep level is often ratios of these or products of music so okay um that's what's really governing what matters not is the parameter isolation and solitude but combinations of those parameters are the factors that match her and maybe you can make one higher and one lower commensurately lower and you're going to get the same result so for example we saw last time for your beta times C and they always appear together you can examine doing a sensitivity analysis of bait on one axis C on the other it's a big waste of time exploring a two dimensional space but for really what's a single dimensional thing you know just betta see you could just name this give it a different name call it D and very d and it's a single dimensional exploration that would be needed so with dimensional analysis we have a principled way of reducing differently by a modest amount but sometimes you know that's significant the number of quantities that we have to deal with in parameter variation okay um insensitive analysis we can also use techniques which seek to instead of exploring all possible combinations they judiciously choose values that are make certain guarantees so instead of examining all possible values of these four parameters we may guarantee well at least each particular value but given parameter gets examined at least once so if there's a sudden sharp drop-off once beta point three um it guarantees that you know beta equals zero be trotted equals 1 equal to Unbeatables 30 try each of those it doesn't guarantee that all that each of those we tried with all combinations of other parameters but i'll be trying them at least once so we might examine we might capture these effects of just that bremer a second thing something called an orthogonal array can anyone describe to me what North ah gamma rays okay fair enough um fair enough um so in an orthogonal array context um I wish I could lead a sofa right now at korbell's in 205 day on in our carbonyl right context we may have several parameters that need to bury um I want to ask Alex to label these phones that's up you know maybe maybe this first parameter is duration as a trial came the second parameter is a number of contacts per day during the search perimeter illness you uh and we may have a prohibitive number of possible contagion of values these are continuous quantities even with a discretization we may have way too many for us to realistic examined so what orthogonal race will allow us to do in contrast to what hyperfuse which will guarantee that we examine italy one combination for the lowest possible value of l one with the next level off etc is one combination orthogonal arrays guarantees to any pair of possible values of any of these any of these particular of these columns of these parameters will examine so we'll examine every single pair of the possible L and C will examine every pair possible of Cena Delta every pair possible of Delta and Mel for example now that's far short nucleus network dimension fraud of the number of all possible combination but examining all pair gives us a certain measure of confidence with captured interaction so in a braver variation context we can use constructs orthogonal rays which are widely used in some other areas like software testing and we can use it instead for imposing on a certain guarantee on earth or level of exploration of the space any logic does not provide mechanisms for these although some others offer will provide for some of its for example Benson provides from Latin hyperfuse um uh let's let's talk however about another off which is more theoretically as a deeper theoretical basis and that is what it's called Monte Carlo analysis how many people in here have seen lectures on Monte Carlo approaches okay um monte carlo approaches are are an extremely powerful probabilistic mechanism for um for insight and you'll find them throughout many areas of computational science um and I will just met make as a matter comment that you will find if you start going into the literature you'll find that you can do actually amazing things with um with randomized algorithms you wouldn't believe possible in other words there are many times where you would think a carefully crafted judiciously chose an algorithm that's deterministic has got to yield better results than a an algorithm that's based on chance he's done right so it turns out Monte Carlo approaches on which make use of certain properties are associated with certain statistical properties associated with with drawers from distributions turn out too often up and keep for certain tasks they can open Pete readily the best no deterministic analysis sometimes adding randomness into your problem can actually allow you to solve your phonecalls sound strength sounds brute force sounds impossible but it's true and there's some interesting theses that have been done explicate amiss in showing just how much power you can get out by sometimes adding randomness in um but in this case are our needs are going to be a bit more prosaic but we'll see that Monte Carlo announces actually allow us to dodge as it were dumb kids it work on the curse of dimensionality we are less affected by that curse the hex we are we're unday we are able to evade that the heck's that would otherwise be put upon us on so here what's going to happen is we're going to feed in probability distributions to reflect uncertainty about one or more parameters and then we'll run the money the model many men cars where many is hundreds thousands often occasionally for test purposes may do it forgot small dozens of times for each such realization the model uses a different dropping probability distributions um so it will draw different values each time from four values of the parameters from distributions and run the simulation model record the results draw another set of parameter values from these distributions run the model and what emerges from this is a resulting probability distribution for model outputs now the interesting phenomenon here is that for a given amount of computational Africa an effort Monte Carlo techniques typically exhibit far more favorable properties on in terms of scaling with with dimensions to a number of parameters then does dealing with a greater value so you might think that look you know solid systematic exploration of a grid taking a grid maybe it's a three metal grid here and you know will impose um values for each value of ceti alpha this so hell maybe for along this axis we have the sea here we see and then along this other axis Delta different values of Delta um so here we have a three-dimensional structure they call a three-dimensional brig on caper grid um and you might think while systematic exploration of the surely would we do you board information for a given amount of effort or guilty more information because your sis matter well it turns compared with scattershot Monte Carlo very good just picking points at random within this grid and sampling at does point how could that possibly be better I mean after all you can be having some points that close to each other at some points you know you're going to be neglecting whole areas of the grid um we were to compare those what could be forgiven for thinking that the grid-based exploration is manifestly better manifestly more robust well it turns out that that intuition is dead one demo it turns out Monte Carlo sampling terms of exploring this space in calculating the properties will often to a much better job of eliciting the required information then we'll give a simple um now part of the intuition here is is arguably the most important part is to understand the scale so ladies and gentlemen let me ask this so supposed to buy a and suppose each trip suppose I D dimensions here to be D different parameters so I d parameters okay and suppose each of those parameters has and suppose I call it in 10 forever um number of processes most each parameter has call it P choices it may not be obvious but it wasn't obvious to me as an undergraduate but it turns out certain letters have conventional uses to them and it's kind of accommodation impolite and improper and seemly that you things in areas outside I'm use K P P as a quantity is normally continuous and it off it connotes probability values k is a quantity that's often used to do so let's use K each of these has k possible choices I know this isn't obvious i wish someone had told me as an undergrad these things because I think I use I use symbols with abandon in my youth and um haha ok so how many different possible exhaustive comedy since we have and different for every piece with k possible values good God how many do we have yes yeah um now those are good choices names and that's okay so Kayden here um so that's the number of exhaustion things it turns out that that all pairs incidentally will be vastly below it it would go up this good it would go up more square so the exact formula is more complicated but so we have cave in the end if we exhaustively explored this space right the mix up k to the end points within this grid if we exhaustively explored that rises graphically with that aniston abreu a friend right how does Monte Carlo sampling rise with the number of mentions we if we want to pick Monte Carlo samples and we sample from within this space how does the amount of work that we have to do vary with the number of samples most we have and the samples um how does the amount of work very we are to nerve no so we could have a huge dimension we did a very small dimension matters a little bit in terms of you know the vector has to be longer and we'll have the example but it is pretty much in them in the margins compared to what often is a big computational effort is computing the value at that okay so running them all so we've Monte Carlo sampling now if it's this performance Rises event it's invariant to in other words it doesn't rise with the number of dimensions this rises rapidly with the number of dimensions and hearings the number of dimensions d um capital d okay um ah but I don't end up taking part of it um so here we have a situation where geomap that we have geometric explosion based on the number of dimensions call it an over there whereas we launched Carlos sampling it doesn't Ross and what this leads to is a situation where you are very very favorable advantages pesotum sue with Monte Carlo sampling so if you um if you want to do something like a sample the average across the states you want to do that's sort of the most trivial statistic you can imagine I want to examine these parameters and want to take the average across all model responses to it if you want to do that it turns out on if you have a limit amount of computational effort it pans out in different ways for these different versions monte carlo sand you just a few around apologize either's do change dimensions it doesn't really matter in terms of how many samples you take as he change dimensions here the number of if you have a fixed amount of computational effort as the number of dimensions number of parameters raw party how would you keep the amount of work constant what would you do to sort of keep the amount of work founded as you as you raise the number Bramley's sorry yeah exactly he would you would sample less than some more sparks so instead of sampling every point one you sample every pond to kind of right and it turns out that that's the fundamental gotcha because it turns out that as the number of dimensions rise very quickly you start to use incredibly pork strips to capture um to capture the underlying statistics we're with Monte Carlo sampling here in good shape no matter how many dimensions you have and it turns out that it scales very very favorably and it's just sort of classic central limit theorem argument for sampling the mean of upper normal distribution for example the variance goes down as one of the square root of n whereas in the case of a of a of a grid you have this sort of situation where the accuracy depends as one over C to the two of them so it turns out that another way to view it is the Monte Carlo sampling naturally spends a lot of space and certain regions that are that are more more probable whereas in this case where the course of grid you're exploring all areas systematically even though certain regions are often less likely that applies less in this situation but it's a general feature with Monte Carlo sampling so Monte Carlo techniques here can be used to essentially attack the curse in motion once you start getting more than a certain number of something like 45 parameters if it comes it can become quite advantageous to start using multi color in the air just see you sir on so it will be determined by the amount of computation combined with so the idea here is the point when we have limited this was truncated limit amount of effort that we can expand on an amount of of computational resources that we can expand um then we have to Shepherd that resource and really there should be a constant here this is sort of implicit in this data people seem theta notation before um have you seen a poem occasion before o of n theta is Theta combines elements for those who are familiar with this oh and Omega of n so basically this states how quickly this rises both from above and below with stating that is it's well bounded from above and below by this approximation of this approximation um whereas oh no tation says arises vista's it even nope no faster than this beta they the other side obviously the homemaker from the other side but this is stating essentially that the error associated with taking a mean of the entire space varies with this and this is some constant and and the theta notation basically ignores any constants in front so interested this is how its scales it doesn't say what the constant is that makes sense so what I'm saying is it's the number of dimensions rise become very favorable to use much problem techniques so what are these Monte Carlo techniques look like in any logic they look like this okay so it's actually quite quite straightforward to implement at a basic level so you can you may have noted before that you can specify four parameters that they're very dim sum range or you can specify we've called free form values for them okay by very the range I mean they vary systematically among a set of values and grid alternatively you can specify a free form expression and this can draw from a distribution now in this case it's a value which should be non-negative the average illness duration non negative value um it would unclear soon on so here we take a draw from a normal distribution with this is almost um I I found that the normal distribution in any logic when I looked it up recently I don't know if it's always been this way you're only but it really thought disturbed me it actually takes these the standard deviation first me with which is going to deeply unsettling and statisticians you know I don't think they throw over bolts but they you know they might be may be pretty unhappy fun because by to mention you always put the mean person not very that's very unseemly um and I'm going to see if I can ever check what's weirder is that the lognormal takes in analogic also it takes the log mean first number one standard deviation so and one is just log in the oven so that's really nice that explains yes indeed indeed so what this is doing is it's drawing from a normal with mean 15 with standard deviation five and that is taking the max is your why is it taking the max of zero insurance that deposit so normal exhibits non-compact support its it goes out to a penny now mighty the tales are very the decrease rapidly so it's very very unlikely that you'll go of too far but there is a chance that it'll be the draw from Rome will be natural and so the max of zero on that make sure that you don't put something less than zero now a week we could do this for multiple the expression and then run it and it will be drawing the values of these parameters from the distributions um and we could draw them from a uniform distribution but more common that we draw them from distribution centered around like in which case we'll be exploring those likely values the areas of the distribution sort of joint distribution will be exploited that it more likely we explored a lot in samples well where it's things that are very very unlikely will be sampled much golf where it's with the grid we're just brute-forcing sport okay so in short you can do monte carlo basic Monte Carlo explorations in this sort of way by this sort of freeform in a parameter variation experiment implementing the appropriate parameters there so uh based on that you know we could we could run this for every run it will be drawing this value of illustration from this distribution and out we get a distribution of values as shown by Monty politicking mr. now so at a given slice here again remember that okay so um the key thing to recognize those there's some plumbing because I important to be the case this parameter end and I have a whole lecture on this I can I should sending the video of seeking or recognized but fundamentally what's going on here in this interior for the calibration discussions as well fundamentally what's going on to your ladies and gentlemen if that this experiment of running main multiple-time beef broth from the distribution it runs named for those sucks the result so and the scarves me start to you since May goes back and runs me and again with the different cell bodies cut those out incorporates them into the data structures that live in the experiment let me say that again it stores the results from running named after sucking them out of stores in the data structures that live in experiment now we have the experiment as an as an instance of the experiment class up way out lives main mains come and go empires rise and fall all within the run of the single experiment and and for each of those each of those mains it's harvesting data that it stores in the indie information in the experiment before the experiment was kind of a pastor for us but here the experiment wrongly sets the values of parameters in Maine but it actually controls their lives it sorta choreographs the whole the whole episode so how is this working HUD assisted her hope how would a mechanistic level is this working well it turns out that if you go to the Advanced tab associated parameter variation experiment you'll see a bunch of places where code I and some code goes before the initial term set up before each experiment run for your simulation run etc what's going on here is that before the experiment runs we reset all the internal data structures we're going to run the a given simulation with a set of parameters and when we run after its run we are going to take the results from that experiment from that mate from me following so following that run maintenance done its job and it goes back to the experiment says yes sir ok what do I do now and this code is going to essentially extract this this data set which is where main historians values from Maine this route is reference to me it's the root class with in Maine you'll notice that each experiment has a thing that may active object class reap it allows you to say for this experiment what main class should it use so we refer to that as root within the context of this experiment code so it's extracting so if it takes its data structures it runs main thing instead creates an instance of Maine bronze it now it's done it extracts the data its accumulated within me it puts it into its own data structures ladies and gentlemen and then after that it can dump that instance and it can go on to the next okay so this allows it to iterate through make instance of Maine after instance made for the different values of the parameters um and this data that it's accumulated it is nothing other than the data that's that's accumulated accumulated here so let me see if I have um ah type a lot of models um okay um ah no no we don't want to look at that um okay uh okay um so I may have to actually go go loaded and I'm if I may have knocked it out for one of the other values but excuse me um I want to do example models here there we go okay um and we want to go to agent-based SAR agent-based calibration there we go oh oh it says it's already open okay I move may have missed it okay there it is it's down here um okay below the one you can't see you shouldn't see um okay so where is this where is this data going here this data infectious 2d where it's going is if you go but there's a thing called data infectious 2d here this is a histogram 2d object which defines a set of horizontal intervals and vertical intervals associate bin so in other words when you incorporate this data and pinned up incorporates it into a set of bins pumping the number of runs that fall into bins within a certain bin representing the time unit and another bin representing the values of this thing that's going in so these define the bins um turns out you can display things in envelope sort of bathroom as well and and then this information ends up being used for this chart ok so this data set infectious TD is used as the basis for this chart that so so in short this day comes from is refreshed by the data that is extracted from Maine and just to complete the circuit let's go to Maine and we see that it's infectious d s is a data set here which records the number of infectious people over time in a way that's automatically updated periodically at every two time steps and this n infectious variables in fact recorded by when people get infected they record that their their infectious actually I think its own entry to this point so in short main maintains his data structure the so-called data set which is a very useful construct for you to record values and and then as Monte Carlo 2d histogram this this overarching experiment extracts the data from it adds it to its own data structure which outlives the instances of main and then the instance that instance of main can disappear because we're accumulate as data here and that's being displayed in this chart that makes sense as to how that works and you know this sort of this sort of experiment can be added in as a privileged sort of experiment here for you to go ad this you could see this a parameter variation six bro okay um anyone notice this is a root class we specify that we can change it later ok so that was some about parameter variation makes sense okay um I've much more detailed I'm trying in this class to go lighter on the on the clicks and clacks of the use of anyone i have i think i've sent you the references to her I a huge amount of material online which is very training oriented come this you do this hold on this um for how to do these sort of things and you and I would encourage you to look at it if you get from us but you folks are at a level of computational sophistication that go beyond a lot of my audiences and so I'm trying to concentrate on the things that won't substantive and one more offer more confirm or long-term value rather than the vagaries of the interfaces the interface is significant allows you to accomplish some things quickly and that's not to be dismissed but what's more significant is when use one clips or feature or another why you why you should be doing this under what conditions this will be useful or that will be useful etc and that's what I'm trying to concentrate on okay um so sensitivity analysis um I should note that a big point of sensitivity is agents a big there is an issue of sensitive with respect to the start of them all the state at the start when you're running the ball you may be uncertain about some of the parameters but here sometimes uncertain about the state as well and in an aggregate model on you you can sometimes very assumptions about this very easily because you know if you have three state if you have three stocks in your mom you can divide up the number of people between you know the different stocks with two parameters one for fracture 114 up to the other have to some too market you know in the in all three of the simple ones all three stocks and totally starting to conflate now agent-based model estate is a far larger too much now an agent based models the the underlying state space is just vastly larger than fern aggregate model for an aggregate model we may have a regional SI are for example an age-based model and principal we have a vastly larger estate space state of all individuals talked about this before the number of states in each individual x right number but the fact is we only occupy off in a very small manifold Reutimann space too small sub space um but if you want to pick a starting state for an entrance mom need more than super there's tons of possible initial starting configurations so often what you end up doing is you end up saying well we'll assume a certain fraction of these characters certain practices and we'll abstract away from with whether it's Mary or Sam or John you know who has those characteristics that that will be assumed to be not socially so just be aware that that specifying the initial state of a van HBase models often more involved okay so it just made comments on sensitivity analysis any questions on this before I just make some introductory remarks yes yeah yes ok it's a good question on yes I'm kind of combining two things with on kind of mixing mixing two producers a little bit um yeah so so what ifs assuming is that we have a leave some probability distribution that we can characterize or over the space in terms of the light we can explore it as a grid and essentially weight each component of the grid so we get a value for that for what how the model behaves for that those values of parameters and then we weighed it by its likelihood of a current and we sort of sum up and there were essentially performing a numerical integration doesn't work um over the sort of an expected value as it were over the outcome of the model alternatively what we can do is just draw values from the distributions of these parameters and just sample the model dirt before each of them and take the mean of each and it turns out that in both cases both of these will be affected by how well we can hug it or distributions in other words it's not like it it is it's an extra imposition for Monte Carlo or or that we ver monte carlo we depend on that whereas for for spread in both cases if we have probability distributions we depending on follow cases having some sort of description of what we think the appropriate probability distributions does that make sense um good question it's often to be um so you asked a question that I was musing about and I don't I have a hunch that that it actually still will be desirable but it's not because of these characteristics and um i would i would ask the paper of a um because there's still going to be favorable properties associated with just sort of the raw sampling of it I mean here what happens is the grids become for certain course refers of course with Monte Carlo samples it's true you're going to be doing it over a uniform grid without concentration on highly likely regions but you will have the ability to to sample in a way that isn't inherently adversely affected by the courses of the grid I think it may end up depending on the shape of the response function in other words on how the value of the simulation model will vary over the space and how steeply it varies etc um and that may be in the play is the choice between these two quick I want to think about this in my mind is clearer because I think it's a really interesting issue and you use zeroed in on them that done that the key the key question that I was musing about enough gurkinn so I I much value and it is one on which writes on news ok other question on montecarlo technique we will unfortunately not be able to talk about in detail in this class of likely unless we make some really really good progress but there's a quite advanced now that is just coming to the fore right now which combines essentially elements of fairly fairly sophisticated statistics with a simulation model with dynamic modeling and it really integrates the two um work we're practicing this here um and it's not why they practice yet in the world but it's it's worth noting and that has to do with parameter estimation of model so we're next going to talk about and so we're going to come back to this issue of Monte Carlo techniques very briefly when we talk about this issue of calibration I want to give you a a glimpse of this and it turns out monkey Markov chain Monte Carlo techniques will come to the fore as an alternative to to sort of calibration looks like looks like i may be dealing with an old version of this but it will be sufficient for showing showing this year so the edge true of calibration is that we're going to again have a friend we have a parameter space here get a conceptually a space of combinations of different parameters of their there LC and Delta here then Todd you and beta and the deal is folks that often and I ok so i should have a header slide of this that that Orient's it so mod calibration comes in falling off on formulation it comes in to help gain estimates at parameters okay um we talked about sensitivity analysis but the issue is that sometimes we don't try to estimate for ammeter galleys we don't have per se on the given parameter but we may have information that's not per se about one parameter or another parameter but it's more about the emergent behavior tomorrow we have observation maybe it's a system or interest and it oscillates with you sort of matter maybe it exhibits those sort of waves we saw on that are involved some infections for example dupla great maybe it's a situation where you know in some situations it takes off and it some it dies out these are patterns and when it comes to age based modeling the types of patterns we can appeal to or much richer yet because they involve then they would be saved for an aggregate system dynamics model because they they involve spatial components and they involve topological components of their networks and often we have observations of a lot of patterns but we may not have enough specifically collective out pics of parameter value to direct the estimate so the goal with calibration and in fact the goal with these this alternative markov chain monte carlo technique will talk briefly mode is to leverage this other data it's to leverage data ladies and gentlemen on emergent properties of the system properties that almost by definition cannot be reduced to just one parameter value or another we can't immediately see that pattern say oh this parameter must be suffering the most inception set there there the results of interactions within the mall between a large number of different elements process and as such they encode information about many parameters but they tell us and if we have large number of these patterns aren't even a modest number they may collectively those patterns may collectively allow us to triangulate the values of parameters they may have allow us to artfully deduce okay what that ranges these parameter values must be in either individually or together so that we see all these different results okay um so some parameters may not be directly observable but there may be some observable data that depends on those parameters that exhibits regularities based on those parameters that we can articulate with a done hmm um so here we may be able to essentially use some of the emergent behavior to try to do what the values of promoters are now with the relationships are more straightforward we can do what's called backing out and we can use we can use data from high-level information to go down to data on more more on detailed parameter voice and I have some examples of this um more generally we're dealing with emergent behavior we're dealing with situations where we have maybe lots of empirical data this is vertical did from whatever almost and we have them all in this case in system dynamics model which seeks to match that data over time and on in order to arrive at these sort of matches we needed to estimate parameters in this model that were less moment and we did it so that the model would best match all these different cases in reserve data so this is merchant Ana you can't just read out the value of any one parameter from this historic day that's over with these sort of crinkly lines and said what we do is we try to find values of model parameters that best match this data now and markov chain monte carlo techniques watch should take moving the pro will actually try to derive protocol posterior distributions over the values of parameters so that we instead of getting on a single best estimate a point estimate as it's called the parameter values a single estimate for beta single estimate for seeing you seem less meant for help that will that we feel best matches these data these emergent theater instead will get out distribution distributions of possible values that's what the Markov chain Monte Carlo techniques will would give us but for traditional calibration we are trying to adjust parameter values and this sort of space we're trying to find where in this space of a particular value for B a particular value from you and a particular value for without here maybe there's somewhere in the space with those particular values where the models match to the observed data is that and that will we will pause it that that's at least a plausible value since it accords with all these different sorts of data now that's not the end of the story what we try to do typically is trying to do this against most of it or half today and reserve the other half and then we see if the model can predict the other half so instead of trying to over fit another words thrown out trying to force fit it to all the data we have and say well that's best gas we try to match it to a lot of the data and then see if we can explain another lot of the data by itself without being told about and if it can that's a suggestion that is robust it's it can predict things that we had no Italy and that ladies and gentlemen is called prosperity cross because we're sort of training in a one set as we're calibrating it on one set trying to find missing values of parameters using one set and then we're crossing over and comparing it against another set of data and that is nothing short of a modeling best practice um but for most of our discussions of calibration we're going to focus on how do we get it to match this and this will be an optimization problem how many people have seen optimization courses perform okay so if I said the words objective functions or payoff function or penalty function or energy function ok yeah um we will be we will be trying to judiciously explore this space so as to minimize our equivalent of the energy function of the penalty function which will be measured by the discrepancy between what the model predicts should have been the case and what we actually are and by doing so over this space and exploring it in a not a brute force sort of way but in a carefully chosen sort of way we will try to arrive at parameter values which that around which we have at least some which we believe have some face validity some plausibility associated at least they don't lead to big discrepancies with data we know when Kirk leads the case i'm going to close now but i love remark that this process is simultaneously what is most challenging on muscle warning processes because this is where you learn often what you will found it may sound like overfitting if they sound like just trying to you know getting it to maps video what you have to find is that your model structure has a lot of regularities that imposes implicitly and it's not like statistics where we describe the form we assume a logistic just or we assume a log linear model we assume a linear model in linear regression and we impose that this is situation where we're specifying the rules for the model whether it be an agent-based model or system dynamics model we're specifying the rules that implicitly imply behavior you know we're specifying the derivatives of the state fairs both the state variables over time but the derivatives up with or we're specifying the state charts and how people interact and then what actually happens over time at a high level is merely implicit in it's merely emerges from it and what you will find is when you are trying to calibrate that your model often tells you something important it will for example refuse truculent obstreperous way but well it may sometimes seem that white buddy doesn't localizes it it will often resist it will be unable to match the observed data and you will push it you will pull and just can't do it it turns out models have regular to structure determines behavior and sometimes you want it to behave in this way and there's no way it's going to be me but it's in this process of calibration you figure out why what is it that is going on here and you learn about the underlying dynamic civil situation you learn oh ok this is not captured I've represented this into simple away or you know now this is really this is a real tension we need to gather more data sometimes and I found this in several cases models that I've had refused to calibrate I've been given data from very very good sources reliable sources and I have tried as best we can to calibrate mulligans and it cannot do it go back and forth we examine all-tournament apart from just won't do it we go back to the data sources and we talk to you look data up and they pulled us all the date as well for this place that had thought that was good reason to talk because for those years we have been it didn't match up because the model captured the situation better than in other cases you go back and say oh yeah back then you know states are measured differently and later was no other way and that's really encouraging because again it's a model predicting something you know it is telling you something and if you actually look into it it can be more reliable than the data you hold up his goals so this is where often learning takes place learning about the mall structure learning about the data and learning about why or when that you can that often gives you the biggest insights improving them all it keeps you honest ladies income this is not about getting the best fit for a model and just declaring the day done this is about this is how we learn about the world more effectively because it's often that emergent behavior we want to explain and often we will find that we could leverage all sorts of bottles and none of them fit in it collectively those patterns imply something deep and this is where we learn that deep thing this is our chance to learn learn that deep underlying structure that that data is telling us and we'll find that we may start way off base but will gravitate towards a model that's much more grounded a model where we've learned something about the structure the world that was not obvious in at the surface level and that is part of and with those words