Print Heterogenous Us State with airSlate SignNow

Print heterogenous us state

[Music] okay divert it and so this is something that's near and dear to my heart modeling and heterogeneous treatment effects so we're going to take a winding path before we actually get to the the gist of the talk so just just bear with me I promise it's worth it so the the data we're gonna look at today concerns this particular question should we rebrand welfare as assistance to the poor and for most of you who are not from America maybe a more important question is what is welfare so in the United States welfare refers to a collection of assistance programs at the state and federal levels some are means-tested some are not they cover things like cash and wage assistance healthcare food and nutrition housing and utilities so how do we answer these types of questions well we run an experiment so luckily some nice folks from the University of Chicago conduct something called the General Social Survey every two years and they've been running it since the 70s and they asked a broad a range of questions including demographic things and attitudinal things and lucky enough for us since the 80s they've been running this particular question framing experiment so we are faced with many problems in this country none of which can be solved easily or inexpensively I'm going to name some of these problems and for each one I'd like you to tell me whether you think we're spending too much money too little or just the right amount and then one group of respondents they asked about welfare and another group they asked about is this assistance to the poor and then so these two things in the most publicly available data center the variables are called net fair and that fair why and then I also just pulled some other variables that are in the in the survey and also asked other respondents the year in which the survey was fielded the which treatment conditioner which particular branch of the survey they were in their response so one if they answered we spend too much and if they answered any other anything else the party ID so in the u.s. this range is on a scale the response we're asked to put themselves on a scale from 1 to 7 or something like that with Democrat being a 1 and Republican being 7 so strong Democrat on one side and strong Republican on the other side political view similar to party ID they're asked by themselves on a scale of liberal to conservative I think again that was from 1 to 7 also age of the respondent the use of education of the respondent and then this composite racial attitude index which was just a combination of four or five separate questions about people's attitudes toward particular other cultures in the u.s. so if we just look at the the top line data so this is people answering too much for each of the question wordings over the course of several years so you can see it changes a little bit it looks like there's a persistent and strong difference between the the two responses and in 1996 which is where that vertical line is the u.s. passed very a Welfare Reform Act that had a lot of changes but you can kind of see people ask directly about well welfare you know policymaking generally lags public opinion so we see a strong lead up into people answering we spent too much just before the US enacted some sweeping changes to some of those programs so we can of course report on the average treatment effect so that's just the difference in means between the treatment and the control group so in this case how many people answered we spend too much because they saw the other wording over welfare which is just generally what we call those assistance programs in the US so we can do this with deep liar and we get people seeing assistance support generally answer we spend too much far less often than people who are asked about welfare this is probably one of the largest experimental effects I've ever seen so 34.7% we can also just do this with traditional linear regression which is also gives us hopefully the same answer subject to rounding I suppose so that's well and great but what we're really here is what we're really either talk about our heterogeneous treatment effects right so we we just looked at the overall effect but we want to know what is the effect on different you know that's the population average but how does it affect different groups of people so one particular way to think about causal inference and this is probably one of the more popular ones is the Neyman rubin causal models so we imagine that people have potential outcomes and we only ever get to observe one or the other but never both depending on which particular treatment group they're in so in this case y 0 refers to the potential outcome in the control group y1 is their potential outcome and the treatment group so you can see the first person was in the treatment group we saw their response for their potential outcome for y1 the other two respondents were in the control group and we see their response for Y and zero but we're missing information on their counter factual so when they were in you know the the response for the person and the treatment group when they could have been in the control group and then vice versa for the treatment group so in looking at heterogeneous treatment effects we're really interested in in this particular metric the the conditional average treatment effects so this is analogous to the average treatment effect except where it look we're conditioning on particular values of covariates and it's in this particular case we we need to examine covariance that are not affected by the treatment so similar to computing the the 8ee except this time again conditioning on particular values of covariance so just to make this concrete this is the conditional average treatment effect conditioned on that political view scale that I talked about earlier from liberal to conservative so liberal being one strong conservative being seven I suppose and you can see you know the dotted line is the average treatment effect and you can see that people on the liberal side self-reported liberal side of the scale seem to be less effective than people on the self-reported conservative side of the scale so how might we actually model this when we don't have any pre specified subgroups we want to look at I guess going back to this Rubin causal model we can sort of look at as a missing data problem what we're really after are using some of the other covariance that we have to figure out what to fill and what should the question marks be so a real simple model that we can use is just include the treatment indicator as just another covariant and then when we want to reconstruct the condition I have a treatment effect for a particular value of x we just apply that model twice once with the treatment flag as one and once with the treatment flag as zero and this is what that looks like in our so here I fed a random forest to this General Social Survey data and then I've I've run the predictions twice once with the treatment indicator set to a sense assistance and once with the treatment indicator set to welfare and then to get the conditional average treatment effect I have just subtracted the two values so I guess an aside model evaluation so how do we know that this model is actually finding things that are like there in the data and not just noise this is tough to do we can't treat it as a normal machine learning problem because we don't actually have observations of people in both treatment and control but we can we can do some I guess some less principled things like use a holdout set so set aside some fraction of the survey data and see how well the model does at rank order so if we sort all the predicted treatment effects and compare you know decile them and compare the actual treatment effect within the group that the model says are likely to be affected to the group that's not likely to be affected do those actually line up so do we see a bar graph that sort of looks like this another thing we could do is this one's comes from marketing so we can compute the uplift curve which sort of represents the what amounts to like the ROI of using this particular model to target treatment so if we use this model to decide our treatment policy who to apply the treatment to does it recommend the people most likely to be affected before the people least likely be affected so here the the blue line is the model the lift of the model and the yellow line would be random just random selections so that the red line pretty much represents the average treatment effect and good model should be above the the random model and you know of course in machine learning we like one number because we can optimize one number it's a little hard to optimize a graph so we can take the area of this and in a few marketing places this is called the Kinney coefficient I guess that rhymes with Jenny maybe so it's just the area of that graph between the random model and our actual model of interest so now back to modeling approaches another another modeling approach a little more sophisticated and the first one we looked at is just modeling the response in the treatment in the control group separately so in this case we fit one model m0 to only the control cases and again we don't include the treatment indicator just the pretreatment covariance and then we also in m1 we model the response as a function of the covariance for the treatment group and then just like the first model we looked at we run the two models subtract the difference and we have the conditional average treatment effect hopefully and this is again what that looks like if we actually did it in in R so this time two models we call predict on both models and then subtract the difference slightly more sophisticated this one's called the X learning I think X the authors intended because it's cross so this one we're actually trying to imputing to directly impute those those missing potential outcomes so first you know similar to the last mile we looked at we model the outcome as a function of the covariance and the treatment and control group separately then we actually in cute you know we we take the modeled outcome for the treatment group and we subtract out the observed outcome for the control model so that gives us D 0 D 1 and then we actually throw that into another model I realize this is really complicated so I recommend reading the paper they treat it much better than I do and then then we're left with two models that are trying to model our imputed conditional average treatment effect as a function of covariance and we weight them according to a propensity score or maybe just some constant if the treatment was entirely randomized and then the most exciting one of the bunch is the generalized random forest which is I guess right now is still a working paper by Susan Athey Julie tibs sure honey and Stefan bogger this is inspired by Carter random force and they're actually trying to construct an in Samba love decision trees in which they're directly splitting on are trying to directly split on the treatment effect rather than doing it indirectly as we've been looking at so far so this is actually on cran it's called grf and they also took the effort of proving bias and consistency guarantees for constructing trees in this way which makes it a little more principled than some of the other things we looked at and this is what that actually you know solving the same promise is what that looks like in our so we load the gr package extract our our design matrix take out the outcome and then get a binary treatment indicator throw that into this function they call a causal forest that takes design matrix outcome treatment indicator and then we just call predict on it and we can pass an estimate variance in which case we get for each individual in the data set that we're calling predict on we get the estimated variance we get a point estimate and the variance of the estimate so that's all good and in putting T others talk I got distracted and also wrote in our package because I noticed that that code gets kind of long and I didn't even bother writing code for the last one so we packaged up some of those some of the first few models we looked at the interact two on the split one and the excellent added a formula interface you can plug in any any ml or model of your choice as long as it has a function that's analogous to fit and one that and predict makes uplift curves those other various plots that's the link on github and this is what it actually looks like when you use it so our response the variables we want to include and then vertical bar and variable that indicates the particular treatment assignment the data and then in this case we're just using a function a trapper that I wrote just called random forests and now we can plot that called predict on it simplifies the encoding some of these these things up by hand so I guess back to the original question should we rebrand welfare and the answer is probably welfare is a very loaded term in the in the US and all the models I tried fitting to this data you notice there are very few people that respond we should spend less money when they're presented with with assistance to the poor instead of instead of welfare so a bunch of references these slides I'll put up on speaker deck so I guess you don't have to read through all of them but highly recommend a lot especially the the causal forest paper so with that here's some links and I guess we'll move to questions yes Oh with like question frame in generally yeah yeah so the question was do people look at header genius treatment effects in question framing experiments generally right yeah it's there's a lot of surveys like this in the US where they'll ask the same question but in slightly different in different framings but in a lot of cases you can also argue like do people think you're asking them about the same thing when you ask them about welfare or assistance to the poor Obamacare and the Affordable Care Act you could maybe argue our different things depending how they were presented in the media and people's like exposure to pundits and things like that that one is we don't really know unless they're unless somebody does like a follow-up design where they ask him like after presenting the treatment like why did you say this and do you think this is the same thing and this particular dataset nobody had done that yes no single treatment but so the question is where is the heterogeneity is it in the the different treatments or is it in people's response to the treatment and in this case it is in this particular modeling framework it is in people's responses to a single treatment explainable by pretreatment co-variance yeah that works too