Lead nurturing for product management

How to create outlook signature

hello everyone yeah I'll be presenting on the topic experimentation in part management I've mentioned this as part one I hope if there is a more interest in this topic so we can have further parts for this session as well so as part managers we focus a lot on inventing and doing things which you are not sure about and the only way that we can leap into the unknown or do things which are not sure about is by experimenting and our very famous part manager Jeff Bezos also of these does so so so in terms Hoff so what will be the ROI of spending for next 20 minutes on this session what will be their key takeaways that you can take that you can benefit by listening to this session in the next 30 minutes first so has it as a TM why should your own experiments why when they launch features so the key takeaway should for you would be why should I is a TM run experiments and how long should I run experiment for one of the biggest cots running experiments is a time it takes to reach a scientifically valid conclusion so I will lay stress and help you realize and understand how long you should run experiments for to reach a valid conclusion and then once you have got some numbers important metrics how should you interpret those results and tape low risk decisions so let's jump straight away into the session and hopefully you will spend the next 30 minutes of your time into something meaningful so first why do we run experiments as I promised I I assume most of you would have heard correlation is not causation so we have heard as multiple times what we see is that there is phenomena one and there is something as phenomena do so you see that you have changed the color of your Buy Now button from blue to red and you see that there is maybe an increase in conversions it has happened during that time but can you confidently say that the increasing conversion has been caused only because of the change in color they can have multiple other things that could have been happening at that point of time so experimentation a running of scientifically by the experiment is the only way through which you can establish causality the very high confidence and very high probability so in order to establish a very clear reason of change we need to run experiments another big reason for any experiments is to detect small changes as you scale in Port Management as you as you work on products which are being used by millions of users it's very hard to detect small improvements and running valid scientifically wired experiment is the only way to detect these small changes and the third and often underestimated and underappreciated fact why you should own experiments is to detect surprising changes or side effects so the I throughout my career I have done many experiments but I've seen some metrics to change me I never anticipated I made some change in the search results page and suddenly my category navigation page visits dropped and ultimately I dig deep and found it was actually a result because of my experiment so there are there are number of unexpected changes that can happen and running valid scientific cement is only weird we take these unexpected changes so now jump stick now let's jump straight into the world why what is the scientific method and how should we run these experiments and it in a scientific manner so these are some basic steps we should have you should ask the right question and do some background research and have a very clear hypothesis and idea but what we are going to change and what is going to impact then you run test your hypothesis with an experiment you analyze your data and you make your decision so these are some very basic valid steps that you should follow while running experiments but the devil is in the details so let's dive straight into the details so I wanted I want to share this case study huges and a real-life experiment and I then may bring my experience nap deal and I won't be sharing the exact real numbers but it will give you an idea about what I wanted to do and what were how should he run Mary it's a new experience for the insight that I had was that users whose uses Astrid searches during searching converted 2.5 x times more and users who do not so there was a clear idea for us that if we can crease the usage of autosuggest or we will be increasing our conversion rate which is another primary KPI is the product managers careful ecommerce support managers would do anything for getting improvements in conversion rates so our hypothesis was introducing autocomplete for searches will improve our courses excuses and then our null hypothesis we introduced autocomplete for searches will have no impact not resist usage now what is this null hypothesis so in traditional hypothesis testing we have this concept of null hypothesis which means which is basically a way to say that whatever it is we make as you they will be there is a way to say that it will have no impact no effect and then what we do is we investigate whether this is true or not in these times of covert nineteen we would be we are hearing more and more about impacts of drugs and tests and impacts of how drugs are being tested and various users so this is the topic is most relevant at this point of time so so we have this null hypothesis and we then investigate if this is true or not so hypotheses are either right or wrong so we either reject or accept the null hypothesis during a test they can be a case of new like inconclusive result as well which we will cover in next session we have to go into more details to cover that so now I share with you some numbers let's park manjae's LUT number so let's have let's dig deep into this so I have run this experiment for three days and I have these numbers so I have I have these three success metrics I know for users who were in control these are the numbers you just users who were in treatment these are the metrics and and I have run it for three days should we decide it should we take a decision and launch this 200 version of the users or not well we don't have what should we do I believe so at this point of time there are two types of message that we can make we can either reject a true null hypothesis or we can accept a false null hypothesis in science statistics these two errors are called as false positives or type 1 error or false negatives or type 2 errors how should we avoid these so to avoid false positives we have a concept called significance level and in general scientific practice we generally use 5% as a significance level what it means is that if I learn hypothesis is true we would see statistically significant result one in twenty times so if we if he sees a very significant reason we can we can say that it's it is a very surprising and significant change and in that case the change in the primary metric was caused by this change made a variance so we prove causality because of this and that is one of the basic premise of any experiments so how should we prove causality how should we say that if our experiment had the significance level or not so now if I just have these numbers I cannot say with confidence anything so I had give you some more information I so I introduced this concept of p-value which is what indicates significance levels and I tell you these these values of the p-value and Delta is something which indicates what was the change in these metrics so I tell you that there was this this much change with a p-value of two percent three percent and seven percent and as we had studied earlier we use the significance level of 5% so any p-value less than 5% indicates that this change is that is strictly significant that means we are we have detected a surprising change and the and that proves that the Delta impact has been caused due to the change that we made in autocomplete so should we launch at this point of time well we have detected that there is some effect in a target matrix which has been caused by experiment however is this change of practical importance have we expose this experiment to enough set of users then do we know whether this Delta is of practical importance or not or whether this is too low too high we still have some level of uncertainty some level of risk that is not acceptable in a scientifically valid experiment so what should we do so we have this con another significance levels often proper didn't win comes to a rescue which is a concept of power analysis power analysis limits false negatives what it does it helps us if the concept it tells us that how long should you run an experiment on and what should be the sample size or the number of users your experiment should be exposed to so that you can be reasonably confident in the decisions that you make and and you are aware that the effect that you have seen with your experiment is of practical importance we can go into some more details about what is power why is it more important into the next session but as of now what I want to focus is that this concept of power analysis helps limits false negatives and improves our accuracy of our experiment decisions so generally we do this analysis before starting an experiment we have this concept of a minimum detectable effect point for a metric we can choose any in multiple effect we can use historical values of the effects that have been caused by that those metrics and then we do a power analysis which gives us the required number of users our experiment should be exposed to to get a state where our experiment is ready for decision so now I tell you this value we know the values of control and treatment we know the Delta my experiment has run for eight days and I also have run my power analysis in at indicates the power analysis indicates that my experiment is the default decision the p-values are all less than 5% so yes hooray we have reached state four ready to launch we have detected a real impact on as methods and which can be safely assumed to have been called by explain so we recommend running experiments for at least seven days or like one complete weeks weekly cycle for two to also compensate for seasonal changes well we we have seen in most ecommerce as well as any any these two platforms graphic varies throughout the week the types of users very slow to be so safe run time period for an experiment is at least seven days a complete weekly cycle you do a power analysis before starting an experiment and you calculate P values so at when you do all these calculations you reach this state is now your experiment is ready for a decision and then you can based upon the metrics that you see you can either and for this extreme these set of numbers is safe safe to say that we can ready to launch and we can launch this feature to 100% of our users and it is such a success but many times the numbers are not this this good they can be sometimes meant to increase decrease so we will have to take decisions of ingly but the concepts of calculating P values 34 decisions are the same for all the experiments and it's absolutely critical for ending of scientifically valid experiment so that's it that somebody had indeed we make decisions to experiments we run experiments explore hypotheses and I definitely recommend you to avoid you through two types of errors by calculating p-values and doing power analysis running experiment for required duration this wanted to some like end this session by this statement where my true belief is that experiments don't fail you may decide not to roll out all experiments teachers new things and your success so hopefully you like the session you you find decision valid and useful and if yes then there are some other details and we can cover up in a follow-up session later where we can cover common biases to watch out for an experimentation how to plan and run experiments effectively what metrics to chew and not to choose for experiments how to calculate confidence intervals how to take experiment decisions in case of inconclusive evidence so thank you and happy experimenting you