Collaborate on Nike Receipt Generator for Product Management with Ease Using airSlate SignNow

Nike receipt generator for Product Management

hello everyone thank you for taking time to attend my webinar my name is nikhil vadera and today i'll be talking with you about test and learn and the significance of it in the product development most importantly in this age of digital where change is constant now before i get started i want to make sure we all know that my today's talk is purely my opinion not my employer and all my presentation is based on the learning which i have obtained throughout my career working as a product manager this is not something which is endorsed by any entity here's the agenda for today we will start with what do we mean by test and learn then we will discuss the significance of this approach in the development of a new product or maybe moving the existing product forward we will also touch base on what product managers can do within their organization within their team to promote the experiments and then i will bring some real world example of experiment that's going to help us to better understand how experiments are conducted in real world and finally we will take a look at the life cycle of experiments and understand how we can adopt to test and learn approach and start conducting experiments so starting with what is test and learn well in my opinion test and learn is simply an approach or we can say a set of practices which enables you to test your new ideas or any potential solution that you have in your mind and measure the impact of it quantitatively before you decide to move forward in other words test and learn is an experimental approach which lets you test the effectiveness of your idea or hypothesis in a situation where there are a lot of unknowns or uncertainties here these ideas are or these hypotheses are tested in a very small scale which means less risk and the data which is gathered from these tests help you to determine which solution offer the most uh the most value or have a greater impact while at the same time it eliminates any uncertainty or guess work um whether the outcome of these experiment is successful or not the findings that you get from these experiments they're invaluable as it helps you on and guide you how you're going to move your product forward so the goal here is not to conduct a perfect experiment but to make a better decision from the output coming out of those experiments now let's understand why experiments are important and its significance in the product development here are some of the questions which could be a nightmare to any product manager or maybe an entrepreneur questions like is this new feature really going to help my customer problem is my new idea going to work is going to make any impact at all in the real world or as a product manager which feature should i prioritize next that's going to help me to increase the engagement rate or conversion rate of my product there are many more questions like this which are filled with unknowns and there is no easy way to get answer to any of these questions especially in today's world where consumer taste and behavior is changing in a rapid speed anyone who's building a new product or a feature under extreme uncertainty knowing if their idea is going to make any difference or will it have any impact in the real world is always a big question and test and learn approach helps you to make well-informed decisions based on data coming out of these experiments these experiments are key to innovation and in order for you to move forward in uh move fast in in the age of uncertainties and stop building products which nobody wants these experiments provide great insight and help bringing the clarity as it reveals what work and what doesn't now if you increase the number of experiments you increase the number of innovation you produce and that enables you to know more and if you fail you fail fast you learn fast and you move fast now let's take an example let's say you have hypothesis that making certain change in your app uh maybe on the registration or a sign up page it may increase the number of daily consumers sign up by x percentage but you are not sure how this change how the user will interact with this change in the large scale or is this change going to cannibalize any existing feature you currently have or is it going to really work or not these all unknowns can be frightening and the only way for you to tackle and eliminate these unknowns are by simply putting your hypothesis to test and rely on the results to validate or reject those hypothesis this approach enables you and your team to be more learning based team who use fact-based decision making to achieve any objective now let's understand what product managers can do or within their team or within their organization to to promote and to conduct experiments well there are many organization who have dedicated team focusing on learning experiments separately but that should not stop any product manager to not wear the scientist hat and conduct experiments of their own let's assume that not many people or your peers in your team or organization conduct experiments or understand it's important in such case creating an experiment in experimentation culture may take time and effort however you as a product manager now have the perfect opportunity to lead by example you should start small which means conduct small experiments and share your results your test results with others this will promote the idea of experimentation and also help building the momentum by communicating your wins will get people excited about the experimentation however the the important thing here is the critical thing that i would like to call out here is that while you are promoting the culture of experiment and encouraging your peers to conduct more experiment you should also prepare them to feel comfortable with the failure because when it comes to experimentation failure is bound to happen it is important to understand that failed experiment does not mean loss it means lesson learned failed experiment produce results and insight which teaches us something which we we didn't know before it tells us what didn't work so you should be open to the failure and embrace it you should also have an infrastructure in place that should enable you to conduct experiment easily by infrastructure i mean uh tools and software that gives you the ability to build small changes roll it out quickly in the market to a small segment of users and the tools which you can leverage to measure the effectiveness of those change what i'm talking here is the feedback loop feedback loop is what will enable you to start the process of learning throughout the experiment and once you conclude your experiment the feedback loop will tell you the how effective your experiment was now let's take a very uh simple example of uh uh experiments let's say you are a product manager and you are exploring the opportunity to lift the conversion and engagement rate of your product through your website your hypothesis here is that if you make certain change in the hero image of your website you can lift the adoption the hero image is basically the first visual component which you encounter when you open the website with main purpose is to attract the customer attention and your experiment here is to find out what a different color on the hero image garner's more click all you can do now is you can conduct a b test where part of your traffic is shown to original version of your hero image which is a and remaining part of a traffic are shown the new version which contains the change and that is b now by showing the random sample of your visitor different versions of your page you can use data to determine which version of your page is more effective in meeting your goal which in this case is lifting the conversion rate now this is a very simple example of experiment the scenario is very simple and the method of conducting the experiment which is a b testing that is also very well known and simple experiment is way more than a b testing so let's let's take another example let's talk about a scenario where you have an idea to build a brand new product from scratch and i have a perfect example for you and which is zappos.com now zappos.com is an online shoe retailer known for one of the most successful customer customer service e-commerce business in the world and how it all started their their founder nick swinburg he was very frustrated at some given point he was frustrated that he was not able to find a specific pair of shoe that he was looking for in any of the nearby stores or mall the problem here was that there was no easy way for him to find that pair of shoe if he travel to the store it does not even guarantee that he's going to get that pair of shoe there or not from there he got the idea to start selling the shoe online he understood the problem pretty well and saw the great opportunity to address the problem he believed let me say he assumed that if we start the digital e-commerce retail experience of selling shoe people will buy shoe online here his hypothesis was that if i set up a digital retail to sell shoe then people will be willing to buy a shoe online however this is just a hypothesis and in order for him to validate he didn't start it by putting a factory or a warehouse where he stored massive inventory of shoe or distribution center to meet the incoming demand of the online order instead he started by running a very small simple experiment he personally visited few stores nearby and made a deal with them to use their inventory he took a picture of their inventory and put it in his in his website a very simple website he told the store owners that once he get the sale done through his website he'll come back to pick up that shoe and will pay the full price the only objective here of nick was to test the hypothesis that people will buy shoe online at this point he was not concerned about storing a massive shoe inventory or building technology to support his e-commerce business or even building a user-friendly website that all came later after his experiment was successful and he was convinced from the output of the data which he which validated his hypothesis of people buying shoe online this way nick eliminated all the risk and uncertainties this experiment guided him to move forward which started starting a company which later in 2009 was acquired by amazon for 1.2 billion now let's take a look at some of the the life cycle of the experiment well first thing first you start with identifying the problem the problem that matter most to your customer now that's your starting point remember your experiment is attempt to solve your customer problem through the series of hypothesis and it should be tied to a product goal or a vision of a company now for example one of your product goal is to improve the conversion rate and the problem you're facing is that there is a high percentage of customer who visit your website but very small percentage of those customers complete the registration of your web on your website and become a member which means there's a major chunk of traffic that comes to your website but choose not to sign up which is resulting in a very poor customer rate which also means that somehow your product is failing to address the need of those customer segment now once you the problem is clear and identified and it's well defined next thing you're going to do is you're going to form your hypothesis which are nothing but a simple educated guess which you which you basically come up with based on some secondary data on which you have done some analysis or maybe coming out of with from some level of user research that you may have conducted now for a given problem we have which is related to the low conversion your hypothesis could be simply by launching feature x we expect the conversion rate to increase by y percent let's take a look at some of the hypothesis statement we have i have here related to this problem which is improving conversion rate so the first one if we optimize our website and improve the load time by 50 then we improve the conversion rate by 20 now let's assume you have formed this hypothesis after conducting a user research which reveals that some percentage of user got frustrated and abandoned your website and you believe this may be because your website was taking too long to load which is why you have this hypothesis that if you reduce that load time you will be able to improve the experience your customer experience and eventually the conversion conversion rate here's another one if we simplify our sign up page and make it more responsive then we will improve the conversion rate by 20 now this hypothesis is formed based on let's say analysis that you have conducted which tells you that you have customer who clicked the register button and started the sign up process but somewhere in the middle they they abandoned your website they didn't click the submit button so you're looking you're measuring the difference between number of customers who click register and submit and you see a huge variance which indicate that customers are abandoning the signup form and you believe that if you simplify the signup process with less questions that will optimize the customer experience and will result in improving conversion rate let me throw one more here let's say you as a product manager you believe that if we provide the social media login as an alternative to registration form then we will improve the conversion rate by 20 let's assume this idea is coming from design team and you believe this hypothesis is a good candidate to experiment and why not the point is not to stick or limit yourself with one or two experiment you can run as many experiments as you want because if you increase the number of experiments you increase the number of innovation you produce and that enables you to know more and like i mentioned before if you fail you fail fast you learn fast and you move fast now let's let's say you have your hypothesis statement ready you understand the problem statement pretty well before you go ahead and run your experiment you want to make sure you have defined the success matrix for those experiments and most importantly you're recording them so that you can measure it after you conclude your experiment when we talk about defining the success metric take a step back underst look at the bigger picture and see what winning looks like for you and you can drive your you can you can produce your solid kpis from there in this case in our case we're talking about where we are looking to optimize our website to improve the load time with a goal in mind to lift the customer registration then our success matrix could be something like a conversion rate now once you have defined the success matrix it's time for you to design and run your experiment when we talk about designing and designing and running your experiment the three most important component you should consider and be very sure of before you initiate your experiment the first one what method or tool will you use to conduct your experiment for how long will you be running your experiment and finally what would be the sample size of your experiment let's start with the method and technique first now there are techniques there are many techniques available which you can leverage to run the experiment and you should pick the right one that is based on the nature of your hypothesis here are some of the examples the first one is off-brand testing now this is a technique which help company to conduct experiment with minimum risk of their brand equity this is something which is best suited in a scenario where you have a brand new product idea and you want to experiment to validate the viability of that idea now companies do not like ideas but what they have a hard time is on deciding what idea they should invest on what idea they should prioritize same was the case with toyota at one point when they applied this technique of experiment to validate their new idea related to the gas payment they were testing if customers will be willing to pay for the gas from the convenience of their car they registered a new domain created a facebook ad and drove all the traffic to that off-branding landing page to test their value proposition another way of running experiment and probably the most famous one is a b testing and also known as split testing and this method is adopted in a scenario when you're comparing two version of element against each other to determine which one performs better i have another one here which is a method is the mvp aka the minimum viable product this is where a very small version of a potential idea or a product is pushed out to the market to gather the real user feedback mvps are built to test assumption and hypothesis related to the value proposition that product is pursuing and this is exactly what nick swimberg did to validate his hypothesis which was related to consumer will buy shoes online the other two components which are to determine how long you're going to be running your experiment and you would be would be what would be the potential sample size of your test now the goal here is to gather the sufficient data from the test which is accurate and reliable and in order for you to achieve that you need to run your test long enough to make sure enough people participated in that experience in that experiment now here i would recommend that you take a help from the data folks your data scientists and your team they can help you to get that answer or maybe there are a lot of tools available which you can leverage to determine the right sample size and the time period of your test let's say in case you're doing an experiment on your existing product by making some small changes and your method is a b test and for for this case the duration of the test will purely depend on how much amount of traffic you get and based on that you might want to run your test for a few days or a couple of weeks and most importantly you keep in mind that you want to run one test at a time to get the most accurate result now most a b testing tool also uh give you the they have the functionality which you can leverage that can help you to determine the right time period and sample size of your test now once you complete your experiment use the matrix which was picked before to decide what your next course of action is going to be these matrix will be the prime indicator which reveals the result of your experiment from here you will be able to prioritize your roadmap with the feature which was which was the winner or maybe not maybe maybe let's say the result of your experiment was rejected it rejected your hypothesis and if that's the case now you want to know why i got x instead of y and this may lead to another experiment and why not the goal here is not to conduct the perfect experiment but it is to make a better decision from the output of those experiments well um here are some of the key takeaways i have for you guys starting with um experiments which i mentioned they are key to innovation and remember if you increase the number of experiment you increase the number of innovation you produce so make experiment part of your product dna uh this is a favorite famous quote by jeff where he claimed that the success they get at amazon is a function of how many experiments they they do per year month week and a day and another one is experiment and failure like i mentioned they go hand hand-to-hand so you should embrace failure the goal is not to run the experiment or to win the goal is to make better decision from the output produced from those experiment and finally measure everything that matters because if you cannot measure it you cannot improve it here are some of the references and further readings i would like to recommend the lean startup by eric reese so very good book it elaborates on how today entrepreneurs use continuous innovation to create a radical successful business another one my favorite inspired how to create tech products customer love by marty kagan in this book marty focus on giving us an insight how most successful tech companies like amazon google facebook they design develop and deploy the product that has gained huge success across the globe hooked uh the the book by need on how to build a habit-forming product in this book need talked about how successful company create product and explain why do some product capture attention and while others flock and the final one building a culture of product experiment by brian jones this is a webinar which is available on product school here brian talked about the different experimental culture and the and the seven habits involved in them well this is all i have for for you guys today i hope you guys enjoyed it thank you [Music]