Digital Signature Product Survey Made Easy

Digital signature product survey

All right, everyone My name is Navin Iyengar like you said I'm a product designer in Netflix where I've worked for many different many years and different parts of the the Netflix product and so what I'm talking about that I'm talking about everything you used to sign up for Netflix or once you remember Sign into Netflix pick something to watch watch it Binge watch on to the next thing all those things are you know pieces that the team that I work on Designs and develops in just by show of hands who here actually uses Netflix Alright so a lot of you so then you know, Netflix is fun and entertaining and relaxing for you But you might be surprised to know how scientific and rigorous and serious We are and our approach to the design and the development of the product And so I'm going to go a little deeper on that today talk about how we think about experimentation, which really means a be testing And then I'll go into how that's really affected the way we actually design and think about the product but first This is not working so You You I'll keep talking while you go but first I'm gonna start with this guy This is Galileo Galilei. Who is you all know? He's a scientist, right? 16th century in Italy Now during Galileo's time science hadn't changed in two thousand years and in fact science was really philosophy the ancient Greek philosophers had looked around at the world made observations and said this is how the world works and nobody really questioned their ideas all through the Middle Ages all up until Galileo's time now one of the things that the ancient Greek philosophers had said Was that the heavier something is? The faster it falls through the air Now on the surface that seems intuitive, right? If you see an apple fall off a tree it goes fast If you see a leaf fall off the tree it sort of floats down slowly Sure, that seems intuitive now Galileo had a theory that that wasn't true But in order to prove it he designed an experiment. It was one of the first experiments And one of the most famous experiments of all time. So what he did was okay, I'm just gonna use the The keyboard here what he did was that he? He went to the top of the Leaning Tower of Pisa And he took two stones with him. They're about the same size, but they were different weights okay, so they were made out of different materials and What he did was that he dropped them off the Leaning Tower and what do you suppose happen? Well, they hit the ground at the same time. Not what you'd expect to happen Right, if heavier objects fall faster than light object. And so Galileo had disproven in one moment What had been considered a scientific fact for 2,000 years? That's really the power of an experiment now as to why a feather fall slower than a stone Galileo couldn't really prove that if you look in his notebooks at the tire and he said, you know It's probably has something to do with the shape of the feather or the air pushing up on the feather But there was no experiment. He could run where he had remove all the air from the Leaning Tower of Pisa, right? now it's cool is That as technology improves you can run more sophisticated experiments So recently on this BBC 2 show they ran the true experiment in in very dramatic fashion they got a huge rock and a huge feather and they got a huge vacuum chamber where they pulled out all the air and They they ran the true experiment and what he suppose happen Soiler there. They are going to hit the crowd at the same time and so we really appreciate this idea at Netflix that instead of listening to Conventional wisdom or what people tell you is the truth you go you go into the real world You collect behavioral data and understand get to your own conclusions And the only difference is that instead of looking at the world of physical objects in motion We're looking at the world of our customers behavior and customer satisfaction and trying to learn more about how that works And how do we do that? well we run experiments - and we found that this idea of thinking of our work as if it were a series of Experiments really frees up our thinking it allows us to try more innovative ideas and crazy ideas and maybe even stupid ideas but in a small scale contained way first to really understand what the effects are of what we're doing before we invest in building that at scale for 125 million Netflix customers Because we're trying to solve for something that you may be - at whatever app or website you're working on Which is that we wit we have way more Ideas than we could ever actually design and build and ship and so what? Experimentation allows us to do is try just a little bit of a lot of ideas to see where we should invest further now at the core of every Experiment is this idea of the scientific method? You may remember from high school I know it sounds fancy but really all the scientific method is it's a systematic way to learn information about how the world works and not just any information but Reliable information information that you can use to predict the future behavior of the world and so we use the same scientific method a simplified version at least for our experiments - so we're starting with a Hypothesis a hypothesis is just a statement of how you think the world works stated in a way that you can prove it - or false and Then you're running some experiment. This is you gathering data from real world behavior and is unbiased away as possible and then there's some analysis or result phase where you're taking what you observed and What your hypothesis wasn't trying to decide whether your hypothesis is true or false now It's worth noting that both of those outcomes true or false are actually pretty positive Because if you especially in a product development context if you disprove your hypothesis Then you have just saved yourself. A lot of time. You might have designed and built and shipped something That actually doesn't have a meaningful impact on your customers satisfaction and if you prove your hypothesis true also great because you've just opened up a fruitful new area of Exploration that can improve your customers experience And so really what I'm talking about experimentation I'm talking about a be testing a be testing is the only way to truly run an experiment in a product development and design Context and be able to say with a high level of confidence that the changes you're making are improving a customer's experience so when we run into an a/b test We we have a 95% level of confidence That what we're doing is the right thing for the cost of customers that still means that one out of twenty times We might be wrong, but it's the highest you can really get And so let me talk a little bit more deeply about how we a be testing Netflix because it's a little more sophisticated than Galileo's experiment We're running controlled experiments, which means there's something that can call the control experience, which is our systems in production as they are today no change because we want to compare that against whatever we're going to do to improve it and ISO mentioned we're always starting with a hypothesis This is just an example but on Netflix we might say that personalizing the content displayed on the homepage will Cause and more people to retain their Netflix subscription So Netflix is a subscription business people choose to pay us month-to-month and what's great about that is that that is a Very highly correlated with what we think of as customer satisfaction If you're satisfied with Netflix you continue to pay us and it's also good for us because it's some stains our business and helps us make more content and make the product better We have that hypothesis, we're creating a series of variations now these are different interpretations of what the hypothesis is if you think about what Personalization means on a given page that could be the algorithm that drives it that could be how its visually presented All sorts of things and ways to get back at personalization And so what we're doing is actually thinking very broadly about all the different things that we could do There's a lot of art and judgment that goes into what we think We actually what has the most potential and what we actually want to design and build And now this is us running our tests So we are randomly putting different people as I sign up - for Netflix into different experiences this could be millions of people per experience and And letting them try out that version of the hypothesis and so it's important to note here Is that when they open their app and they're in that experience? They don't know about all the other experiences so they act in a really unbiased way now the other thing about a/b testing that I think sometimes people criticize is That they think it takes all the creativity out of design and that's not true at all in my experience What happens is that once you are allowed to do all these different variations? You can try really creative ideas that are very different than each other. This is not about little button colors or pixel level changes and then at the end of some period of time We're determining a winner of our test and that's really going back to these metrics Like I said from customer satisfaction, the more satisfied people stayed with Netflix longer and they were in this experience So once we have that winner that's rolled out to everyone. That's the new default experience of Netflix And that's the new control experience for future tests. We want to run so in that way We're constantly Iterating and changing and building and moving the Netflix product forward but only in directions that make sense because they improve the customer experience Encapsulate that idea and what I've learned from it is that we think of product development as a series of Experiments and I propose that that leads to stronger designs whether you can test or not now I know not everyone in this room has access to a be testing you need a lot of engineering Infrastructure, you need a lot of customers to do that Well, what I would propose is that even just thinking about your design process in this more methodical way can lead to stronger designs Now so far I've been talking about Hypothesis, like that's an easy thing to get to But in reality you may have more hypotheses than you know what to do with and you don't know where to start or you may be too overwhelmed and have no hypothesis at all And so there are a lot of different tools we have in our tool belt that we combine with a/b testing to develop a hypothesis So for example, we're running Ethnography studies. This means we're going into people's homes. Maybe only talking to two to three people a day We're going to different countries like India or Thailand but we're really living with people in their environment trying to understand how they live their lives what some of their unmet needs are and You know, of course, we're only talking to two or three people So that's not representative of how everybody feels but it helps us get an idea of some of the nuances behind things And then of course we're running focus groups and usability studies This is where we're bringing people Into an artificial environment and it might be more people five to ten people a day we could ask them questions Understand why they do some of the things they do Or at least what they say they do we can put prototypes in front of them and observe how they use them and ask them What they think of the experience? But again, this is not representative of what everybody thinks. It's just the five or ten people we brought in that day And we can run surveys. This is even more people This might be thousands or tens of thousands of people that we can ask questions to them online. Right and we're asking their opinion so, of course There's some bias in that but what we do know, is that the more people we ask the more Representative they are of what the larger population thinks at the end of the day though. It's still someone's opinion And of course We also have data we have trends in our own data that we can see and use this is a more quantitative method But it doesn't tell us what hypothesis we should test really all it tells us is how people currently use Netflix So in this case, I could say with 100% certainty that more people watch Netflix on TVs than owns For example, that doesn't really tell me what I should do what is my hypothesis for how you're going to improve the customer experience now that you know that and So if I would could draw a line between all the things I just talked about which are ways to really develop generate and develop that hypothesis Really a/b testing is the only way to run an experiment and validate one of those hypotheses And so we've probably run Tens of thousands of tests during my time and Netflix and the most interesting ones are always the ones that reveal these non-intuitive things about human behavior and the the top line of that is really that what we've learned is that you need to observe what people Do not what they say. And that's what a be testing in its unbiased way of observing people really allows us to do this There's this quirk of human behavior that people say things but that doesn't really correlate very well with what they will actually do in real life And so as an example Here's a survey that we've run with potential customers from Netflix many times We've asked them what one thing Would you like to know more about before you sign up for Netflix any guesses on what this big red? Sector is feel free to shout it out I think I heard it somewhere it was content People want to know what is the content you have on Netflix? You might be surprised to know that when you go to sign up for Netflix today Or at least in the past you weren't able to find out what our catalog was We actually made you sign up for the free trial and then once you're in there You could pick whatever you want to watch and watch it over those 30 days for free and So this idea that that's strange has definitely been percolating up in within Netflix Every time a new designer joins the Netflix design team to say hey, why don't we do this? And so we had this hypothesis, right? The hypothesis is that by showing people what our catalog is before they sign up for Netflix It'll actually make them more likely to sign up So that's the hypothesis that we needed to test and we could have just gone straight to a/b testing there But we wanted to understand again Like I said develop that hypothesis understand some of the nuances behind it to make our designs even that much more likely to win in an AV test and So we did some qualitative research this was our non-member homepage at the time which pretty simple pre focused how to start your free month button some supporting points and Then we built a series of prototypes. I'm just gonna show you one for purposes of the narrative here where we allowed people to Scroll down from there and see all the titles that we had available You could navigate over to any different genre. You can see what horror movies we had on Netflix all the while You couldn't actually play anything. If you try to play something. You still have to sign up for the free trial That's just how our our licensing model works so now when we took this into qualitative studies People said yes, this is exactly what I need to decide whether to sign up for Netflix And so we thought we had a hit on our hands But when we actually observed them in these qualitative studies standing behind the glass and seeing how they use the prototypes We realized that they were getting a bit bogged down. They were getting into what we call internally shopping mode So they were looking for a specific piece of content or title. That's said to them Yeah, I want to watch this so I should sign up for Netflix And so we're starting to believe that this might not be actually the best nonmember customer experience But the only way to go from believing that to knowing that was to run an experiment to run an a/b test And so we did that Actually rn5 a/b tests each time We had the control experience, which was our our normal page without access to the content and we had variations we allowed people to go explore our content in different ways and Every time the winner was actually the control experience experience without any content in it, very non-intuitive Right what the learning here that we found? Was that that's not actually you know Just a page of content is not an indicator of whether you should join Netflix or not when people join Netflix for a free trial There's all these other things that we can't communicate on that page Like the fact that you can pick from all these different titles and they start immediately You can start watching on one device and pick up on another the high quality of the streaming all of those things are not able to be communicated just on a page where you can see what titles are on Netflix and So it's a really good learning for us. Of course. We wanted to listen to our customers. Our customers are still asking us for something which is that they want to know what the content is that have a better idea before they sign up for Netflix and So through subsequent rounds of testing This is actually our homepage today, and I'm sure it may change in the future as we continue testing But what we have here is a non-interactive wall of content that shows up in the background here and what this is it's personalized It's tailored to what region you're in what content is available? So it gives people an idea of what's popular in the region Likely something that is they recognize in here and lets them know that hey You're going to get to watch all these things when you sign up But it doesn't let them get bogged down and trying to find that every specific thing that's in our catalog And so I challenge you to think about the same thing What behavior can you observe if you can if you can a be test sure do that because that is the most unbiased way to observe people but even what we saw in our qualitative sessions, is that when we ask someone whether it's the right experience they may say one thing but if you actually observe how they use it and Try to be as objective as you can you'll see that there might actually be a discrepancy between those two things So if I could just encapsulate that approach again is that we think of product development there's a series of experiments and I propose that that leads to stronger designs whether you can test or not because you can just approach your design in a more methodical That's my time thanks very much You