Print Customer Gender with airSlate SignNow

Print customer gender

don't know you hello honor Henderson assistant professor of marketing currently at the University of Oregon today I'm talking about how to use marketing engineering to predict customer behavior using choice models with marketing and engineering software the logic model the agenda first I'll provide a brief introduction into what choice modeling is and how it's used in marketing applications I'll transition into a personal application that I use to evaluate a loyalty initiative and then discuss the marketing engineering choice model in case that's an assignment in my marketing strategy class so what our choice models choice models are an analysis approach they attempt to determine the impact of different marketing factors on customers behavior they're very popular it's the most popular individual level response model in marketing it uses past behavioral data to predict future behavior there's no need to survey customers you can survey customers and use survey responses as part of your model to predict behavior but you could also just use available data in your customer relationship management database so in furs what is important to driving customers behaviors of most interest to marketers that determines the probability of choices and the elasticity's of marketing variables in terms of driving those choices and then uses a binary or a multinomial logit model basically that means we observe certain outcomes that are discrete so either a happens or B happens and that's the binary choice model or if there's potentially three options or four options or five option we could observe one of those five options happening and try to understand what drives choices amongst options that's the multinomial logit an alternative way of modeling behavior is using our aggression analysis and those we use for continuous outcomes such as the amount of money customers spent with us over the past year sales dollars would be a continuous outcome it gives weights to drivers of the outcome so we could do an analysis of how advertising spending impacts sales dollars but it doesn't give a probability of a certain behavior happening it does not use the logic model so what is a logic model that is used in choice modeling loads of model helps us assess the impact of marketing on behavior and we can understand the marginal impact of a marketing action whether the marketing action makes a choice more likely or less likely that's the marginal impact being higher well and you can see that the impact of a marketing action will depend on the probability of a customer choosing an alternative let's say one alternative is to stay with the company that's going to be retention another outcome would be for a customer to defect or to leave a company if a customer hates you and they're 100 percent guaranteed to leave it won't matter what special royalty initiative you do it won't matter how much you cut price they're definitely going to leave you and if they're for sure happy and they love you and they're going to stay you don't need to do a marketing action to try to convince them to continue to stay with you there's a very low marginal impact of a marketing action towards that customer so you could save your money to target customers who are in the middle so this assumes the customers make rational choices based on their individual utility and there's a diminishing sensitivity to marketing efforts so the sensitivity to market efforts will depend on customers propensity to do one action or another so Marketing has the highest effect on people who are sitting on the fence so if a customer is not really sure whether or not they're going to stay with you or defect those are the customers you might want to target with a marketing action let's compare this briefly to the conjoint analysis so what's in between choice modeling and conjoint analysis both are used to model customer decisions and customer preferences now some contract models do use a logic model they do use own choice behavior to predict how customers value certain options but it's set up as an evaluation and it's set up as an experiment or the types of options that are evaluated are picked through the study design and it's basically done with lots of control the researcher uses lots of control to have certain options evaluated and from those evaluations you can infer preferences so you need lots of customer input especially if you're looking for ratings from customers on different options you don't need as many customers because it uses kind of an experimental design or an orthogonal set of attributes to have certain products be evaluated you don't need past purchase behavior you can have customers evaluate a hypothetical product offering and give their impression their evaluation it is appropriate for lead users it's you know projected for innovation and the cost of doing is relatively high that's because you need to run it as a study choice modeling we just observe behaviors so there's no real survey necessary there's no real high level of interaction with customers we can just see what they do and track them in our database we need a lot of customers to get accurate in Finch's we need to have past purchase data so we're going to correlate past purchases or past decisions to different variables that we can use to predict whether or not a customer will choose option A or option B it's not appropriate for lead users because it's backwards looking you're assessing probabilities based on past behavior and the cost is real well as long as you have a customer database you can take the variables for customers out of that database and match it with customer purchase behavior and use that to try to predict why would one customer continue to buy while another customer would defect why would one customer buy option a and another customer would buy option B so what are some common uses of choice modeling well mainly it's used to target particular marketing actions to particular customers how do you do this well first you want to create a database of customer responses so track customers behavior and ideally track customer behavior over time so what are the one of the choices what are they actually doing and you can do that with an experiment and a control sample so perhaps you want to do a particular marketing action and see how effective it is to driving customer responses you can have one group of customers get the marketing action another control group not get the marketing action so one might get a special discount they could be the experiment group they used to assess the special discount another group of customers might be left alone they might be ignored in comparing those two samples and the choices of the customers across those two samples for instance whether they continue to buy or whether they defect then you can run the model and test whether or not that experimental variable was predictive of customer attention or you can look at prospects you can see whether a certain kind of outbound marketing effort drives customer acquisition if you can observe some customers clicking on your website and other customers ignoring it or some customers clicking through and making the purchase while other customers look at something on your website but don't continue to make a purchase and you have any other variables or information around them maybe you've done an a/b test where you do two different versions of your website you could use that data to run a model that would say hey which version of my website is most effective for driving clicks or driving purchases you could also not necessarily do an experiment but just look at historical data of ask customer purchases so I have a bunch of data on a bunch of different customers some bought some didn't buy somewhere acquired some weren't acquired some bought the special version or the high-end version of my product while others only bought the low-end version and if I have that historical data and I can look at other variables on those customers I can see what makes a customer likely to go ahead and do a certain behavior that I'm interested in so then once we have our database we model customer choices and we can rate customers based on their probability of success whether they're retained whether they're required whether they buy the more profitable version of our product we can look at the response to marketing efforts or our is one marking effort more or less effective than another and if we have forward-looking data we can try to estimate the total customer lifetime value maybe you can look at the likelihood to acquire customer to retain a customer to expand them to get them to do word-of-mouth and if you look at all those choices and you allocate a dollar amount in terms of the long-term profit you would make off that customer given their probability of those different actions you could give them a customer lifetime value score and based on that score you could target customers with the highest success rates and you can target them with the most effective actions and the highest returns now as I said and the slide before when you come targeting customers if a certain customer is guaranteed to be a good customer they're definitely going to stay with you they're easily acquired and you find out that they're very likely to respond positively to whatever you're going to do then you should go ahead and target them with that action but if they're already so positive such as they already love you so much that there's no way they're going to defect there's no way they're going to leave you then you wouldn't spend any marketing actions on them you might save them for your customers that are on the fence as long as that's fair and as long as customers who are your good customers don't see you treating mediocre customers treating them too well and as long as it comes across with some fair method then you can still keep your existing customers that love you and also sway some of your customers on the fence to stay with you okay so let me talk about application well I use the choice model to assess the impact of a particular loyalty initiative so what is a loyalty initiative there are customer specific investments they augment the core offering so the augment just the basic product that's good that gets sold and they aim to protect and expand existing customers business they're increasing in prevalence given the trend towards proactive engagement amongst companies so there's a rise in continuous service providers such as Netflix or even Amazon where you do subscribe and save or different cloud companies business-to-business service as a solution with all these companies oftentimes you sign up your customer you acquire them and once you acquire them you stop engaging with them until a particular problem comes up or they're thinking about leaving you or you're trying to sell them on some upgrade but there's also been a rise in vice presidents and directors of customer engagement customer experience customer success and the thought behind these new positions is that let's not wait until we want to upsell a customer oh let's not wait until they're going to have a problem let's try to proactively engage with them through some loyalty initiative that gets them to use our product or service more it gets them to be happier with it they get to lock them in to their loyalty and makes them predisposed to respond positively if we try to upsell them to a newer better high-end version of our product so these customer engagement experience managers they are given these budgets and they're supposed to go ahead and create these loyalty initiatives and you can use a choice model to evaluate a loyalty initiative and so just briefly I checked the other day on LinkedIn and I found a whole list of companies that we're trying to higher heads of customer engagement analysts for customer engagement director of customer attention all these directors and analysts would probably be doing choice models where they evaluate different customers on their likelihood to stay or be engaged to be willing to buy a newer version of a product and hopefully they'd be using some type of choice model to assess whether or not they're large initiatives are effective the problem with loyalty and this is is they're very expensive about 1.2 billion dollars is estimated to be spent on them annually if you consider the employees that you have running them and assessing them and also just how much it cost to engage these customers proactively and try to make them more loyal it's very expensive and the effectiveness is unclear so previous academic research have said we aren't really sure that they're worth the cost that there's a positive response to outweighs how much we spend on these loyalty initiatives and from that there's still relatively little research informing whom to target so which customers should we send these lo2 initiatives to and so I did a research project working with a continuous service provider to try to assess the success of third loyalty initiative and also understand how certain characteristics of customers make them more or less likely to respond positively to a loyalty initiative and so let me to kind of describe the study and so you can get a sense of how customer engagement employees and analysts and directors can use a choice model to to better allocate their marketing resources and be more successful at their jobs okay so the study design these were the inputs I'll describe them in a second and these are kind of outputs or the behaviors that we observed so in terms of customer performance we observed three discreet behaviors that we would see certain customers do it would either defect these existing customers would defect and leave the company they would maintain the status quo so they would continue using and buying what they were buying before or they would expand they would add on to higher and better versions of the product service offering and so those are three different outcomes when we can look at what's the probability of any customer doing these outcomes as a function of their demographics and also any other variables that we have now one of the variables or three of the variables that we want to look at or what we call intrinsic loyalty mechanisms and so we gave each customer based on some information we had in their customer database a score for habit being a low habit or a high habit we looked at whether they were dependent or not whether they were locked into a contract and whether they had certain bundling on savings and dependence is thought to keep customers from leaving and also a relationship how long have they been a customer and how well did they kind of have an existing relationship with the provider and that's also thought to limit defection so all three of these kind of intrinsic behavioral loyalty mechanisms we had a rating for each customer on all three and we could use them to predict the probability of these different outcomes and for instance when we ran our choice model we said how likely is that a customer would choose to defect choose to keep things the same or choose to expand we'd see that habit made it more likely an existing customer would choose the status quo continue to do what they've done before and less likely that they would make changes and some of those changes the company want to avoid such as defection but other changes the company was interested in seeing customers do which would be expansion and dependence we saw there's a strong probability of it driving a lack in defection but not necessarily it didn't make a difference between status quo or expansion and so you can look at how any driver any variable you have on customers changes the probability of a customer choosing a certain option and so we use the base on a baseline period of five months to get indicators of customers habit dependency relationship and we combine those with indicators of customer demographics where they live how many people were in their family stuff like that to also predict the probability of them exhibiting any of these behaviors in the future but then we combine it with the field experiment and so after our five month baseline period we did two months where customers got to use the service for free but we only had half the customers get that loyalty initiative and we wanted to see what the impact of that loyalty initiative would be on the probability of each of these outcomes would it sway the likelihood that a customer would be more likely to defect or less likely to defect or they'd be more or less likely to kind of maintain the status quo or would they become more or less likely to expand so we could assess the efficacy of this particular loyalty initiative on these different outcomes by having some customers get the large initiative while other customers didn't and by tracking that who did get it and who didn't get it and also observing these outcomes we could look at the impact another thing we did is we looked at if it mattered whether or not the customer had to have it before a dependence before or relationship before and so we looked at the interaction between the experiment and their previous scores on all these different variables and the idea being that maybe if a customer had a habit they'd be more likely to continue to do the status quo that makes sense if they were kind of left alone but the loyalty initiative could potentially wake up the customers it could make them more engaged with the product service category as you know the director of engagement hoped but in doing so if these habitual customers were woken up to consider what they want to do you know consider new alternatives other status quo you might see an increase in defection or expansion as even though they're happier that they got the loyalty initiative they're now aware of maybe I should consider doing a different option than I had done before when my habit first set in and actually that's what we observed with our choice model and so basically we observed what the outcome was you know for an outcome period of eight months for these customers and all these inputs and were able to combine those inputs together and assess what the probability was of these different outcomes and we could say hey you should target the lotion ishutin to a customer that has habit and dependence because the dependence will keep them from defecting and the lotion ship will wake them up and they'll consider expanding and so that was some of the takeaways we had for the company for the customer engagement program do the lower two initiative okay so let's talk about the homework assignment you have with marketing engineering it's a relatively straightforward and simple homework assignment I'll introduce it and then I'll go ahead and show you how to do it so it's the book binders book club and it's a direct marketing exercise and it uses choice model in to assess who will be most likely to purchase the art history of Florence a particular book so according to Oh first of all these are pictures of me when I used to live in Florence so unfortunately these pictures were not in the art history of Florence so that's why many customers did not buy the book but getting back through this homework assignment it includes this quote that says according to Doubleday president Marcus Wilhelm the database is the key to what we are doing we have to understand what our customers want and be more flexible I doubt book clubs can survive if they offer the same 16 offers the same fulfillment to everybody that just captures the idea that customers are different some customers will be more inclined to buy a particular book than others and with that philosophy in mind we can do a choice model to see which of the bookbinders book club customers will want to buy the art history of Florence and we can try to look at what marking actions the company might do to make it so people would be more likely to buy that book in the future so what happened bookbinders book club mailed especially produce brochure the solicited purchases of the art history of Florence so they had all these existing customers in their book club and they sent some a special brochure on this particular book and said would you like to buy this but now sending that brochure you know this mailing cost printing costs so there's costs involved and so you only want to send it to the customers that are you know highly likely to buy it if you have a choice model you can predict which customers are likely to buy and which customers are not so they put together a data set of a small group of their customers and in that data set 400 customers purchased the book in response to the mailer well where 1200 did not in the idea being if we can find out which customers are most likely to buy the book then we can go ahead and target the customers who are most likely to buy we're the amount of purchases will offset the cost of the printing and the mailing of the brochure okay sorry I had to shift this slide around because I had an overlap in some of the text so anyways 400 customers purchased 1200 did not let's predict let's use a model to assess the likelihood that a customer purchased the book as a function of 10 predictor variables that they have in their data set so one of the variables they had they knew the customers gender they knew the amount of money that was spent on bookbinder book club in the past they knew the frequency that the customer bought books over a certain time period I knew how long it had been the month since last purchase and how long the person had been a customer um so how long how many months had been since their first purchase they also knew the number of purchases in five different so there's an art category a children's category a do-it-yourself category and two other categories of books and they are able to look at how many purchases a customer had done by book category from that information they were able to see how these predictor variables related to the likelihood to buy a book so if a customer had was male versus female how would that change the likelihood that the customer would buy the history of Florence the art history of Florence and all these other variables how those variables would impact the likelihood of a customer purchasing this book and so the implications of that information once you run your model and you assess how these particular variables on impact the probability a customer purchases the book is that you can target certain customers that are most likely to buy the book and I'll show you how to do that or you could use it in your forward-looking strategy to try to change what you try to do so for instance if you really wanted to kind of reposition your company around art history books or Florence books let's say these are high margin products for you you might try to recruit customers with a particular gender or you might try to retain customers if that month since first purchase is really important predictor you know depending on what predictors are important you could design marketing actions that would try to influence your customer did your customer portfolio in terms of how they rated on each of these variables and so this last slide I'll show you before I get to the case this is not something you have to do in the case but I just want to show you the impact of using a choice model so you can apply the results to a holdout sample and you could evaluate models and optimize profits this is kind of direct marketing 101 and what I did is I ranked customers based on their predicted likelihood to buy the book and I put them into deciles so I basically sorted customers in a holdout sample based on their predicted I could purchase the book and then I looked at what percent of customers in that decile actually did purchase the book so using the logic model yes with this MN L stands for I see that in the customers who scored the highest on their predicted likelihood to buy the book 37% actually did buy the book and I could also do a regression model on the same variables and regression model performs a little bit worse or only 35% of the customers in the top 10% of customers purchase the book if I add another 20% of customers that were ranked high on their likelihood purchase then I get up to 49% of all customers who ended up buying the book versus with the logic model it captures 53% of all customers that purchase the book and the more and more customers I include based on the rankings eventually I get to a hundred percent of the customers that did purchase the book even though each of them represent a smaller and smaller chunk you can see down here I have the total number of customers that purchase the book based on their probability score that they would purchase the book from our model and the slope is such that at first the more customers I include the more and more customers actually purchased the book but eventually you know these are customers who are predicted not to purchase so very few of them actually did purchase so I'm comparing predictions in terms of their predicted likelihood by their actual behavior and with the logic model it's slightly better for predicting who bought the book then also if I do some assumptions based on like the profit margin taking into consideration how much it costs to mail and how much money I get from selling a book I can see how much money I make changes as a function of how many customers I target the mail into and basically this model says that I can maximize my profits if I send 70% of the customers the mailing and I should send it to the top 70% the 7% that are predicted is most likely to purchase the book and that's what maximizes profit now that's just the straight up cash numbers one thing I want to emphasize in this class and everything we do is beyond these numbers and I love these numbers I want you guys to have an analytical approach to decision making but beyond just these numbers there could be other factors that would make you want to send these mailers perhaps you send it to all hundred percent of your customers and you would be okay with your profit margin on this particular product going down if for instance you thought it helped raise your brand awareness to customers or if it helped reposition you in a strategic part of the market for instance maybe you're very much interested in being seen as providing this type of book whether or not a customer would buy it or not or maybe you want to send it out because you find in a different analysis that if customers get hit with a mailer at one period in time and don't buy it and they're more likely to buy a product in the future from a different mailer and so if you have a long term view it might begin to make sense to target more of these customers who have a low probability of buying so the pure simple look would say hey target about 70% of the customers because you have diminishing returns of this marketing effort sending the mailing but you know there could be other strategic reasons why it would make sense to go ahead and do this so this is the power right here of doing these type of models whether it's a logit model or a simple regression model usually you wouldn't use the regression model for something like did they buy or did they not buy but I just did it to compare and show you that even doing the simple regression model can make you more effective as a direct marketer alright so let me go show you the case and then you can get some additional insight into how do it okay so this is our data set and first you want to orient yourself to the data set what we have is this choice variable which basically says one did the customer buy or 0 did they not buy the art history Florence we have their gender one and zero you'll need to look up what does one mean is that male or is that female and what is zero amount purchased how much money have they spent frequency how many books to the by over a certain time period last purchase how many months sense their last purchase in first purchase how many months since their first purchase and then the number of children books the customer of spot in the past the number of youth books number of cookbooks number of do-it-yourself books and number of art books so looking at all this information we can use it to predict um choice so we use all these variables customer data that we have to predict whether or not they actually bought the book and this data set has been sorted in such a way that all the customers at the top have chosen to buy the book and all the customers at the bottom have not it doesn't have to be sorted in order it's just the way that this data set is set up looking at the bottom I'm hitting shift and arrow down and that highlights all the customers in this data set you can see there's 1600 customers and about 25% of them or 25% of them purchased the book and we can find that out by you know doing it equals average and just summing up these values and I have to go up to the top to find out which number it is at the top perfect so yes 25% of customers bought the book and we can look at other variables so about 66 percent of customers in our data set have the gender that corresponds to number one two hundred dollars is about the average amount of money that customers have spent I forget what all these variables represent but you can get a picture of what's the overall sample you don't need to do this calculation because it's part of the results set when you run the choice model so how do you run the analysis it's super easy at least for this simple model you just go to customer choice load it and run analysis first column contains respondents IDs more than one case per respondent would be if you had customers that made choices over time so you know if you had whether or not they bought the book this month next month in the third month the fourth month of fifth month for instance if you had lots of different on inputs for customers of a choice then you could have more than one case per customer or poor correspondent and more than one alternative per case would be if you had not just do the buy did the not buy but did they buy print a brand B brand C well in this case we just have they they bought the book or they didn't buy the book generate alternatives specific constants you can do that or not it doesn't really matter I don't necessarily want you to interpret them but that just basically says how likely is it a customer does not buy versus how likely is it that a customer does by significance level in terms of looking for significant predictors of whether or not a customer bought the book typically we want to know things that are significant at the 5% level of air so if it says it's significant and we can be 95% certain that the predictor actually drives likelihood of choosing to buy the book or not buy the book you can also perform an analysis on the holdout sample if you have a holdout sample to answer the questions in the case you do not need to look at the holdout sample but I did post one on the blackboard that can help you see how you would use the results in a direct marketing campaign so don't click that and late in class options is if you want to combine this analysis with the segmentation analysis and the segmentation is based on like how much the amount of purchased impacts choice so it uses the relationship between these variables and choice to categorize customers and segments we can't do that in this case because we only have one response per customer but it's pretty cool that this software allows you to do a segmentation with a choice model if you have the right data and then next steps it just asks you where your data is so we click Next we make sure that it's highlighted the right data range the right cell range so we just scroll down and make sure it's all captured and it is so we click OK and now at this point it's very easy we just go ahead and look at what are the results we try to interpret the results and try to infer what we can do as a marketer with the information we gain so response probability is just the probability that a customer would go ahead and buy the book so for this particular customer they said it was a 14% chance that they would buy the book and given that they predicted that this customer would not buy the book but an actuality the customer did buy the book the observed response was that they did buy the book the dummy is just what the predicted dummy dummy dummy all those are just the opposite it looks at the likelihood that a customer would not buy the book so if you wanted to frame these results in terms of the negative then you could use that information these columns to kind of describe what happens out of 400 customers so there were 400 customers bought the book and of those 400 we actually we accurately predicted 160 of them so we could do equals this divided by that so we can say we were set right no way oh I did the wrong number so we were able to predict the customers would actually buy the book correctly 40% of the time that's not an awesome number but if you look at predicting whether they did not buy the book let's see we got that much more accurately and so yeah so I have the 1200 that didn't buy the book we were basically write 93 percent of the time I would have to change it 93 percent of time and so the the key value as a direct marketer is basically if you ignored these 1120 people you won't make you won't waste money on mailing them or quest to buy the book if they're not likely to buy it and so that could save you a bunch of money and as long as you made a high enough margin off the sales of these 160 people then you know hey hey that's great you can cover the cost of mailing it to 240 people that don't buy what we can look at here is the averages for each variable this is the overall total average so about 60% of people had their gender as being a number one and then you can also look at how these averages change by whether or not they bought the book or whether yeah whether or not they bought the book the response or whether they did not buy the book the dummy no choice you can see that some of these variables differed by quite a bit or others didn't differ that much and from that you can kind of get a brief sense of whether or not those variables are predictive of choice but those aren't the most kind of accurate way of understanding the relationship between the variables and choice the most accurate way is based on looking at these variable coefficients and also at the elasticity's now let me talk through those real quickly so if something is highlighted either red or green that means it was significant it means it we can kind of have a lot of confidence that these variables predicted the likelihood the customer would either buy or not buy the book and so we see gender being negative and so that means if a customer had their gender being a 1 let's say that was male they'd be less likely to buy the book if they've spent a lot of money in the past then they'd be more likely to buy the book in the future now these are just saying how do these kind of variables relate to likelihood to buy the book and whether or not they're significant but to interpret the size of the impact the size of the prediction we want to look at elasticity's because these numbers are swayed based on the variable that was used there are very few people that had purchased an art book or do-it-yourself book these numbers were typically between like 0 and 3 where amount purchased was up in the 200 was the average amount purchased in the past and so the coefficient is very small for amount purchased and comparatively it's very big for these other variables and that's not really a fair comparison because it's not normalized to kind of standard unit elasticity's are normalized basically it means if this variable increased by 1% then what would be the impact on the likelihood a customer would buy the book the response and it doesn't make sense it's a gender increased by 1% but you could say gender increased by a hundred percent and so you could take that and multiply it by this fair but basically when you get to these percentage relationships you can say that it's a normalized value so you can compare how gender is different from amount purchased and in this case gender has a higher value and an absolute value of 0.27 then amount purchased which is 0.2 won so a change in gender is more kind of predictive of choice than the amount someone spent in the past even even though they both matter and they matter in different directions if someone is a 1 then they're less likely to buy the book and if there is 0 for gender and if they spend more in the past if they have a higher amount purchased and they're more likely to buy the book in the future ok we talked about this estimation already or it says what's the probability of response I'm going to show you now in the holdout sample how to calculate this value and so let me go to my holdout value without sample okay so basically I copied and pasted have to move this over so I copied and pasted the variables and their coefficients and I'll show you how these values relate to the predicted likelihood a customer will buy the book so I I move these coefficients to try to correspond to these values so gender had a coefficient of negative 0.86 amount purchased I the coefficient of point 0 0 1 9 and what I can do with this holdout sample is I can see how useful our model is for predicting behavior and a holdout sample so I observe their choice I observe their gender the amount purchased I observe all their data on all these variables and what I can do if you go ahead and look at this spreadsheet online you can see the formula first you look at their attractiveness this is just a name of a score using a logic model it's basically the exponential of the intercept plus the coefficient for each fair time's the value for each variable so so the coefficient for D five times the value in D seven that's what you see right here and you add those all up and take the exponent the exponential of the number and gives you the attractiveness score and if you take that score divided by one plus that score you get the logic score which is two predicted likelihood the customer will buy the book and I did a little equals if function here and I said if this value is above 0.5 then say that there a one that we predict they will buy the book and if it's below 0.5 we'll make it a zero and so then I do an accurate and I compare their actual choice first their predicted choice and I can see if that is the same I give myself a score of 1 say my model is accurate and if it's a score of 0 I say it's inaccurate and so I can do a lot of in from I can do a lot with all this information I could do a sort or I sort customers based on their logic score basically their probability of buying the book and I can go ahead and target customers and the highest probability and I can compare if their predicted choice and their actual choice corresponds so I can get kind of a hit rate by deciles that was what I showed you in the PowerPoint slide so if I sorted all these customers by the loads of score and I grouped them unto deciles and I compare its decile their predicted choice first their actual choice I could see how accurate I was and go ahead and assess at what point should I stop targeting customers with the mailer and I can also do some assessments of how accurate my model is by comparing if the predictive value corresponds to the actual value all right that's all I have for you on choice models I hope that you found this to be an interesting and illuminating video that helps you see the value and using choice models to understand why customers exhibit certain behaviors and then from that you can look at okay as a marketer how can I use this information to target particular customers or change my marketing efforts to redo what I do as a company and also who my customers are such that I'll be more successful in the future okay I'm out of here have a great day good luck