Discover the Best Online Bill Book Format for Product Management

Online bill book format for Product Management

... Jeff would say, we took it as an article  of faith. If we served customers well,   if we prioritized customers and delivered for  them, things like sales, things like revenue and   active customers and things like the share price  and free cash flow would follow. So therefore,   when we're making a decision thinking about  a problem, we're going to start with what's   best for the customer and then come backward from  there. That informs what's the work you have to do   to then create this new solution for customers. Today my guest is Bill Carr. Bill is the co-author   of the book Working Backwards, which is a  synthesis of the biggest lessons that Bill and his   co-author learned from their many years at Amazon.  Bill joined Amazon just five years after it was   founded, stayed there for 15 years where he worked  on the books business, and then as VP of Digital   Media, launched and managed the company's global  digital music and video businesses, including   Amazon Music, Prime Video, and Amazon Studios.  After Amazon, Bill was an executive in residence   at Maveron, an early stage VC firm, then chief  operating officer at OfferUp. And these days, Bill   runs a consulting firm called Working Backwards,  LLC, where he and his co-authored, Colin Breyer,   help growth stage and public companies implement  the many practices developed at Amazon.   In our conversation, we go many levels deep  on how to actually implement a number of the   practices and ways of working that helped Amazon  become the success that it is today, including the   process of how to actually work backwards, how to  organize your team with a single-threaded leader,   how to divide up your metrics into input and  output metrics, how to practice disagreeing   and committing, how to implement the Bar Raiser  program in your hiring process and so much more.   Huge thank you to Ethan Evans for making  this episode possible and introducing me   to Bill. With that, I bring you Bill Carr,  after a short word from our sponsors.   Today's episode is brought to you by Assembly AI.  If you're looking to build AI powered features in   your audio and video products, then you need to  know about Assembly AI, which makes it easy to   transcribe and understand speech at scale. What I  love about assembly AI is you can use their simple   API to access the latest AI breakthroughs from top  tier research labs, product team to startups and   enterprises are using Assembly AI to automatically  transcribe and summarize phone calls and virtual   meetings, detect topics in podcasts, pinpoint  when sensitive content spoken and lots more.   All of Assembly AI's models which are accessed  through their API are production ready. So many   PMs I know are considering or already building  with AI, and Assembly AI is the fastest way   to build with AI for audio use cases. Now's the time to check out Assembly AI,   which makes it easy to bring the highest  accuracy transcription plus valuable insights   to your customers, just like Spotify,  CallRail and writer do for theirs. Visit   assemblyai.com/lenny to try their API for free  and start testing their models with their no-code   playground. That's assemblyai.com/lenny. This episode is brought to you by Coda.   You've heard me talk about how Coda is the doc  that brings it altogether and how it can help   your team run smoother and be more efficient. I  know this firsthand because Coda does that for   me. I use Coda every day to wrangle my newsletter  content calendar, my interview notes for podcasts,   and to coordinate my sponsors. More recently,  I actually wrote a whole post on how Coda's   product team operates, and within that post,  they shared a dozen templates that they use   internally to run their product team, including  managing the roadmap, their OKR process, getting   internal feedback, and essentially their whole  product development process is done within Coda.   If your team's work is spread out across different  documents and spreadsheets and a stack of workflow   tools, that's why you need Coda. Coda puts data  in one centralized location regardless of format,   eliminating roadblocks that can slow your  team down. Coda allows your team to operate   on the same information and collaborate in one  place. Take advantage of this special limited   time offer just for startups. Sign up today at  coda.io/lenny and get a thousand dollars starter   credit on your first statement. That's  C-O-D-A.io/lenny to sign up, and get a   startup credit of $1,000, coda.io/lenny. Bill, thank you so much for being here and   welcome to the podcast. Thanks, Lenny. Thanks so   much for having me. Pleasure to be here. It's my pleasure. So, I was reading your book,   and something that I recognized as I was going  through this is just how many new ways of working   Amazon contributed to the way tech and business  runs. And I made this little list, and I'm curious   if there's anything I'm forgetting that's obvious.  So, obviously the idea of working backwards,   the idea of one way and two way door decisions,  the concept of disagreeing and committing input   and output metrics using memos versus decks,  just the idea of two pizza teams, and then I   know that evolved into single-threaded leaders.  Is there anything else that's just like an obvious   core thing that's maybe almost too obvious that I  don't even think about that Amazon contributed?   The one that's non-obvious and is really  the way in which Amazon created a set of   leadership principles that were very real,  and the way in which Amazon created a set   of processes to reinforce them. I think  I certainly haven't encountered anything   quite like that. It was very intentional.  So, that is also a distinctive element of   that we try to point out in our book. Awesome. Okay. So, maybe we will come back   to that, because that is also really powerful  mechanism. So, the question I wanted to ask   about this is there are companies that are bigger  than Amazon, that are more successful than Amazon,   that have been around longer than Amazon, but I  don't think any other company has contributed so   many unique, new ways of working and also been  able to coin them into such shareable ways. What   would you say it is about Amazon that enables  this sort of way of working and also just making   things so just proliferate through the culture? That's actually one of the reasons why Colin and I   set out to write our book because everyone knows  about Amazon as a innovative product company,   at least certainly during the time I was there,  which was from 1999 through the end of 2014.   The company rolled out all kinds of innovative  products. The Kindle, AWS, Alexa, Echo, the Prime   subscription itself is innovative and... And it's all those things, by the way.   Yes, a lot of people around the world  use all those things. And obviously,   Jeff was a huge driver of those things. But what  people don't realize is that Amazon was actually,   to some degree, equally focused on process  innovation. In many cases, by the way,   we stood on other people's shoulders, we cannot  take credit for having... For most of these,   there were other inspirations or we built on work  that others had done, which by the way, was what   I think all great companies should do. And again,  that's also why we wrote the book was because we   would like to allow people to stand on Amazon's  shoulders to learn what we learned, and then take   all or part of these things and build from there. But to more directly answer your question,   how or why did this happen. So, this period  of both product and process innovation   actually occurred in this one narrow window  of 2003 to 2007. During that window of time,   all of the products I just mentioned and all of  the processes except for one were all developed   in this one four year period. Wow.   And this is the period actually where we were  going from hypergrowth stage, zero to one company,   to what I would call one to whatever, a thousand,  infinity. That next step that companies have to   make where what happens is things become very  complex. We're no longer just a bookstore,   we sell a lot of things. We actually  branched out beyond just a retail business,   we had a third party marketplace business. We were  experimenting in those days with providing running   websites for third party retailers in those days  too. We were developing new things. We were in   many countries around the world. So, we'd become  very complex. And what happens to that point is   that then you reach this point where the CEO can  no longer be in every important meeting, can no   longer be involved with hiring every person. And  you need a system, a method to run the company   effectively. And Jeff Bezos is fundamentally,  he's a very scientific and analytical thinker.   His undergraduate degree was in computer science,  I'm pretty sure. Although I think he actually   started off wanting to get a physics degree,  he ended up moving over to computer science.   He spent his early days at DE Shaw as a quant  on Wall Street. Very quantitative mind. So,   he applied this... When he thought  about this problem, he said, "Well,   I need to be scientific about this. There needs  to be some system or some approach, some mechanism   for me to be able to manage such a company. So,  I'm going to experiment, like a scientist would,   with different ideas, different hypotheses,  implement them and see what works, and iteratively   improve." So that was the mindset which we took...  Which by the way, we applied both to process   innovation, but also product innovation. Awesome. I had Eric Reson, and he also   happened... I thought about this at the same time,  he contributed a lot of core concepts to the way   tech worked, and he actually brought up a couple  concepts that were on the cutting room floor,   basically things that he thought would be  things people adopt everywhere. And I'm curious,   is there an example of that at Amazon where  you built a process and had this clever term   for it and just never spread or never actually  worked at Amazon? Anything come to mind?   The dev team, the design team, the product team,  they're all in one group, and they'll go operate   autonomously, but not completely autonomously  because we, the senior leadership team, Jeff   and the S-Team want to know that they're on the  right track. So, we're going to create something   called a fitness function, which was let's figure  out what are the four or five or six metrics that   matter most for your particular area. Let's give  a weighting to all of them and then let's create   an index for those, and we'll measure that index  up and down. And that's the fitness function.   That is a very nerdy way of  organizing teams. I love it.   Yeah, super nerdy. But we realized after, I don't  know how long, several months or a year of doing   this, so the fitness function was not a good idea.  This is what I would describe as a compound metric   where you try to take several important metrics  and munge them into one. The problem is it's   actually becomes totally meaningless. When you're  measuring things, you're trying to understand what   actions or reactions are creating the good outputs  that you want, revenue, customer growth. But by   putting them all together, you basically obfuscate  that. And what really we realized is we need to   just break each one of these out individually  and manage them each in its own way. So today,   I discouraged teams and companies from  creating any sort of compound metrics.   I've done that once, and it was a terrible  idea as well, where we had six different   metrics and every quarter, we were going to  move a different metric that contributed to   a higher metric. And what we realized is we  just never learn how to get good at one thing,   and then it turns out there's always one thing  that actually impacts the bigger goal most. See,   you just end up working on that thing anyway. Let's actually go deeper into the single threaded   leader piece since you mentioned it. It's actually  come up a lot on this podcast of people working   this way where they have a single threaded  leader. And so clearly, it's worked. And I   guess we'll just help people understand what does  a single threaded leader actually mean, and then   why is it such an effective way of working. So, the concept of single-threaded leadership   was first... When I was born from this time of  complexity at Amazon, and where again, large...   Once you get to a certain scale, you get to a  point of where there are competing departments,   competing interests, and they're competing for  some centralized pool of resources. For all of   you who are working for a tech company, this is  this pool of engineering resources, or today,   data science and AI resources. There may be  other constrained resources often designed   as a constrained resource. But the point is now  all these teams want that pool of resources to go   build stuff for them, but they're in competition  with each other. So, most companies solve this   by having an intense, centralized, highly  collaborative process. We decided to go in the   other direction for the reasons I mentioned, which  were that we're just fine, that we're spending all   our time in these meetings, planning, and a lot of  the work we were doing, the artifacts we create,   the documents, the projections, we're actually  not very useful either, we're bureaucratic   time wasters largely because a lot of the  assumptions built into them were deeply flawed.   So, you're debating numbers in these documents  that are based on flawed assumptions, which is   a waste of time. So, what we realized instead  was how do we get... The three things we really   wanted were ownership, speed and agility.  And so, we experimented with that and said,   "Let's create teams that can stand alone, where  there's a single leader and the cross- functional   resources that they need are all either  directly report to them or are dedicated   to them." So they don't necessarily have to be  a straight line direct report. In Amazon's case,   for the most part it was. There were some dotted  line, but it could be all straight line, it could   be all dotted line, it could be a mix of the two.  But fundamentally, we've moved from what we called   a project orientation to a program orientation. So, a project orientation means, oh, we are going   to do this project to change our search result  page and algorithm, and the project is defined   in this way and it's going to take six months.  The resources will come and swarm on that,   and then they'll move off to some other thing in  some other part of the company. The program based   orientation says, let's stick with the search  example. There's a team that works on search,   and they always work on search. And instead of  thinking about things on a project by project   basis, they think holistically about what they  need to do to improve search. They have a set of   metrics by which they're looking to drive those  metrics largely. Ones that they can control.   Things like what percent of the time is a customer  clicking on one of the top three results in my   search page, or how many milliseconds does it  take for the page load time in this browser type,   on this device type, et cetera, et cetera. And they then are running their own roadmap.   They're deciding what are the most important  things for us to go work on, and having a   prioritized list of those things and be able  to start at the top of the list and work their   way down with the pool of resources that they  have. Sometimes, and most times, they may want   more resources to be able to tackle more, but they  spend less time in resource contention, resource   fighting, and instead, focus on building what they  can build with the resources that they've got.   And so, the benefit of this is if there  are success or failures, they're really   dependent on themselves now. The only thing  they could maybe argue about how they could   do better is if they had more resources, which  they can petition management for. But this way,   it also solves a big management problem, which  is instead of management, senior management   refereeing every item on a roadmap, they're  refereeing which teams have how many resources,   which is more of like a once or twice  or three times a year decision versus   refereeing everything on the product roadmap.  And then all the resource contention issues,   that's a daily issue. And so, it frees teams up  then to actually go and sprint ahead. There's a   lot of work you have to do to get ready for this.  For example, in a software environment, when we   first started and we had a monolithic code base  that was not pretty, we weren't ready to do this   because you have all those interdependencies. Once we moved to a service-based architecture,   and then teams could own their code with defined  endpoints, APIs that other teams could understand   that are well-documented, then we could move  in that direction. And the other thing is we   had to create, what I would call, countermeasures  because there's no free lunch in org structures,   any org structure, you're trading off one  thing for another thing. In this case,   you're trading off potentially functional  excellence. So, in other words, if you   no longer have every single engineer or every  single marketing person or every product person   or every biz dev person reporting into a C-level  leader of that particular function, and instead,   they're spread out in small teams across the  company reporting into some generalist who is   probably not going to have functional expertise  in several of the functions that they're leading,   you risk the problem of then the people in  those teams not gaining functional competency.   That's the downside. And we can talk more about  this, but we created a lot of countermeasures to   still enable us to have functional excellence  while creating these single-threaded teams.   To drill into this a little bit further, is the  origin of this, this recognition at Amazon that   the best stuff comes from one person's  vision and just one person driving and   one person's ask being on the line versus the  often, the decision by committee approach?   It is less about that. I do want to be clear,  it's one leader and their team who are accountable   and responsible. So, with respect to what are  we going to go build, how are we going to go   measure success, all those things, this team and  that leader are responsible for documenting that,   writing their plan. Now, they don't just get  to go off and do that. There was an intense   review process at Amazon where either at some  level, whether it be the vice president, senior   vice president, or all the way up to the Jeff  level and his direct reports called the S-Team,   this plan would be reviewed and scrutinized  deeply as well, and there'd be a discussion,   an interchange, and basically getting alignment  between the senior leadership team and each   one of these single-threaded teams on that  plan before the team could go off and run.   The beauty of that though is that once we'd had  those discussions, those interchanges, then the   teams were free to sprint hard after their plan.  They didn't have to worry about whether was,   "Am I aligned with my CEO? Am I aligned with  my senior vice president?" They could know that   they were. But yes, this creates then clear...  If they're going to deliver it or not. It's up   to that owner and that team. Whereas when you  have this highly cross-functional approach and   there's not one clear person who's responsible for  this one project that's on this roadmap, I've seen   many CEO pull their hair out saying, "I have no  ownership and accountability here. How do I have   that?" They're pushing on a string. Because they  can't because their different people and leaders   are part owning, half owning a long list of things  instead of fully owning a short list of things.   I like that. I like metaphor of pushing on  a string. Is this approach similar to just   the GM model, or is there a big difference  when someone's thinking about going GM model   versus the single-threaded leader approach? Yeah. Obviously, there are probably different   definitions of what people consider the GM model,  but I would consider that being this person is a   P&L owner, and you can, of course, create mini  P&Ls within a P&L. Like for example, in the book   business, we could, and I don't know most of the  time we didn't do this, but we could have created   a P&L owner just for fiction books, or just for  professional and technical books, which is a very   large category with big differences between the  others. And then you say, "Great." Then that team,   they have their own dedicated team. They're  fully responsible for the revenue numbers and   other numbers. But you have to be thoughtful about  how you do this because one of the three questions   you have to ask when you establish one of these  teams is, does the team have the resources within   their control to effectively manage this part  of this department, this product, this P&L?   And sometimes then if you narrow things down  too much in some cases, then the answer is   no. In other cases, the answer can be yes very  easily. A great example, this was in Prime Video,   one of the businesses that I managed, we  could create a single-threaded team who   just was working on applications for TV sets,  like Samsung, Sony. We could create another team   that's working on game consoles, and another team  that's working on mobile phones and tablets. And   then within each one of those, we could further  break it down. We could have one team working   on Xbox and another one on PlayStation,  another one just on iOS. In those cases,   then it's very clear how you can break the teams  down and they can have very clear ownership.   Awesome. Let's go back to the countermeasures  topic, and then even just a little more broadly.   You talked about one thing that was important  to put in place before you moved to the single   threaded leader model, which is creating  APIs, and basically breaking apart this   monolith. What are some other things that you  think you need to put in place to be successful   in trying to shift to this model? The other thing was these functional   countermeasures. So, let's stick with the  engineering, for an example. So, in 2004,   2005-ish, I started managing a single-threaded  team. Actually managed two different ones,   one for music and one for video, which are now  Amazon Music and Prime Video. They weren't called   that in those days. But I started managing a small  team of software engineers at that point. Well,   I have never... Well, I have written lines of  code, but that would be back in high school,   and we're talking about Basic and Pascal. I  have a master's in business, a background in   marketing. I'm a generalist, okay? So, I'm not  equipped to coach. I couldn't possibly conduct   a code review. I couldn't possibly conduct an  architectural review. I couldn't possibly coach   or mentor an engineer on how to improve their  craft. But I was one of many of these examples.   And there could be reverse examples where  instead of me being a business leader,   I was purely an engineer, and now I'm managing a  team that does marketing and business development.   I wouldn't know anything about those things if  that had been my background. So, what we did,   and I'll stick with the engineering examples,  we came up with various countermeasures. One   example was that we still had a C-level  leader of engineering in Rick Dalzell,   and most of the core infrastructure and core  services still reported into Rick. So-   Core services still reported in to Rick. So it  was things like payments or infrastructure search,   and Rick still could be a technical leader  for the whole company and he and his team   could create things like what are the standard  ways that we're going to do code reviews? What   are the standard ways across the company that  we will interview and screen engineers? What   does the promotion process look like? What are  the defined steps getting from an SD1 to an SD2,   SD3? How do we document and describe what are the  requirements? There are many things like this.   Effectively, what it also meant is that  anyone who is an engineering vice president,   or in many cases a director, they would often have  something else beyond their day job of some sort   of subject matter expertise area where they would  also contribute to the company. A good example of   this would be that they might sit on a panel for  promotion from a certain level to another level in   the engineering world, or they might be available  to do code review outside of their organization   for another organization. So people had other  jobs in addition to their day job to build and   maintain functional excellence. There are a lot  of examples like this across the company.   Let's go in a different direction and talk  about one of my favorite principles of Amazon,   which is disagree and commit. I think in the  way I even describe it I know is wrong. I think   people hear this term and they often use this  principle incorrectly. For example, it actually   starts with have backbone and then disagree and  commit. So I'd love to just hear how you've seen   this actually implemented well and what people  should do and think about when they're trying to   implement something like this at their company. So when I was at Amazon, there were 10 leadership   principles and they've since expanded them.  But of those 10, this was always the least   well understood when I was at Amazon too, and  partly because it is actually the most nuanced   and difficult to actually use. So here's what  it means. What it means is that have backbone   and disagree, meaning when we are making any  kind of a decision, important decision, if you   are part of that team, part of that unit, it is  your obligation to voice your point of view if   you disagree with your approach that's been taken.  The point of that disagreement, by the way, is to   provide usually additional information or a new  point of view that people have not considered.   So I like to geek out a bit on the process of  decision-making and have read more and more   about this. I think that Peter Drucker probably  has the best writing on this topic. But as he   would describe it, good decisions are made by  first understanding all the different points   of view and pros and cons to the potential  issue at hand or the potential direction,   and that great leaders, what they do is they  solicit these different points of views. They have   a team that they work with to debate and discuss  things. So another way to think about this,   a king and their court. In an ideal world, if  you assume that there's no political motivations,   the court is there to advise the king and  help them think through different problems and   provide different and opposing points of view to  allow the king to arrive at the right decision.   This is sort of no different than that which is  the disagree part is about bringing forth new   information, new data, new point of view that  would be contrary to the current direction. So   that's the disagree part and you're obligated  to do it as we would describe sort of all the   way up the chain if necessary, if it's an  important issue and people are not hearing   or understanding your point of view. Now  the important point is first of all about   hearing and understanding your point of view. What would often happen, I can tell you if someone   in a leadership role, someone come to me with a  disagreement and many times I'd appreciate it,   by the, way because they'd bring some point of  view that was useful, but sometimes they bring   the disagreement and cite the reasoning behind it  and I already knew that reasoning. We'd already   thought of that reasoning, we already thought  of that, in which case I would say, "I hear your   disagreement. We have already considered that  factor. But even though that factor is there,   here are these other factors that outweigh that." Now that is the point at which as long as the   disagreer is hearing back from the leader  that they understand their point of view,   understand why they are pushing back and seem  to fully understand it, and they've taken that   into consideration, that is the point for  them to commit. Because the point is you   provided your information, they've processed  that information and they've decided to go   this way with the knowledge of that. Where people  get confused about is they don't maybe understand   when they're supposed to stop disagreeing is  one thing, and so hopefully that explanation   made people clear this is when you're supposed  to stop, and then the commit part done well   means that it's not just like I'm going to  commit, I don't really agree with what we're   going to do, but I'm going to get behind this. Ideally it's, oh, now I've heard the argument,   I've actually now thought about the argument and  hopefully that person has now understood why we're   taking that direction. So their commitment is  based on that understanding because then they   can reflect that understanding back to their  organization too. Because the worst thing to   do is to say, "Yeah, we're committed to this. I  don't really agree and I still think it's wrong,   but I'm committed to it." That's not actually  commitment. This is really about decision-making   and understanding the facts and information  that people are going to use to make a   decision and then be able reflect that back. I imagine there are many times I've gone through   this where I still don't agree. What's your  advice to a manager or to a report of just like,   okay, when you actually still don't agree,  how do you behave? Do you just behave like,   yes, I agree with this and don't really  voice your concerns or something else?   I work with Jeff on all kinds of different new  ideas. Jeff doesn't think a normal person. His   level of sort of creativity and the way he  thinks, the timescale of which he thinks,   there's many ways about the way he thinks that  there was no one else in Amazon that thought that   way. So there'd be times when even after we've  had that discussion, I would maybe still disagree,   but then what I would do is I'd focus on, okay,  well what is the kernel or the core of why Jeff   thinks that we should do this and I would focus  on that kernel. I got great advice actually from   one of my managers at one point, Steve Kessel  who said, "You have to look for what that is,   and then your job is to then take that kernel  and try to run with it and expand it and try to   see how I can take that idea, that concept,  and then make it into something viable."   It doesn't always work, but it's about then having  that understanding of what it is, not just sort of   going through the motion of stomp, stomp, stomp  through it. That's not going to work. Also,   I've seen people who try that and their career  doesn't go very far. You have to have some degree   of faith that there's something there and I'm  going to try to do the best I can to make that   part. How would I productize that idea? How would  I make that viable from a business point of view   or whatever the different constraints are. Awesome. So the advice there is focus on the   parts you agree with and think about how you  can find out if it's actually right or not.   Agree with, or even just you may not even  agree, but what is the core of what that   person is thinking is the big benefit or good  guy or thinking vector that they're on that's   causing them to want to go in this direction. Thinking vector, love that term. Along the same   lines, another principle that I love is leaders  are right a lot. I feel like this is a term   that it almost goes unsaid. You almost can say  this in a lot of companies. I'm curious just the   origin of why that became an important principle  and then how it's implemented at Amazon.   Yeah. So going back to this last discussion, so  one fallacy we should all acknowledge is that when   you're making these decisions, and you're trying  to use data to make decisions, you can make the   data kind of look however you want it to look to  sort of try to meet your decision. If I'm looking   at some issue and I've got some big dataset, I  can come up with ways of looking at a dataset to   support this idea and ways of looking at that  dataset to not support it. So the data rarely   makes the decision for you. What is happening is  then a lot of judgment and interpretation of the   data, weighing that, weighing various  factors to then come to a decision.   That is sort of the right a lot part. The  right a lot part comes from having what we   call sort of sound judgment, which generally  come... Some people maybe are born with this,   not a lot of them, mostly they get it through  experience. A lot of experiences actually about   being wrong, by the way, about making mistakes  and by having looked at a lot of problems,   made decisions or observed others making  decisions, being a student of that, and   then using that to understand then how to weight  different information when making a decision.   So right a lot is that you're good at that and  that then it proves, and that generally speaking,   people want to follow someone who ends up by and  large going in the right direction, right? You're   the leader of a team. The team is petitioning you  on multiple sides. If you keep kind of going off   in some direction where most of the team is  scratching their head saying, "I don't think   that that was the right decision," they're not  going to want to follow you very far and you're   probably not going to go very far. So this is  something that you develop through experience and   I'd say from having the opportunities to observe  and work for others that are good at this.   I love that it's a lot. I like that it's not  just leaders are right. It's right a lot.   Yeah, yeah. No one is right every time.  That is totally unrealistic. Yeah.   Let's talk about the titular concept of your  book, and that's a word I've never used,   but I think it's appropriate, which  is working backwards. First of all,   just what does it actually mean to work  backwards versus working forwards?   The title of the book comes from two things. One  is one of the leadership principles, which is   that customer obsession, and the principle states  something along the lines of that great leaders   start with the customer's needs and work backwards  from there to sort of meet those needs or solve   them. Then also because we created a process  in this window I was talking about earlier,   the 2004 to 2007 window, we created this process  for new product innovation called the Working   Backwards PR/FAQ process. They both refer to the  same idea, which is that as your guiding star or   the point from which you're going to start  is what are the customer's problems or what   are the customer's needs, and then figure out,  okay, well what would be the solution to that,   what are potential solutions to that? To do those things, starting with without   the constraints of my financial constraints,  my resource constraints, my legal constraints,   my engineering constraints, whatever all  those constraints may be, because the   problem is what most of us do is we start with  those constraints and work forward from there,   or we start with things like I got to increase  revenue. How do I increase revenue? I need   to increase active customers. How do I increase  active customers? For customer oriented behavior,   we tend to start with those things which  may often lead you in the wrong direction.   Whereas we had, as Jeff would say, we took it as  an article of faith. If we served customers well,   if we prioritized customers and delivered for  them, we took it as an article of faith that   then things like sales, things like revenue and  active customers and things like the share price   and free cash flow would follow. So this  is important because I still can't give   you objective proof that that is true, I don't  know who could, and so it was saying this is   an article of faith that if we do that we  think those other things will work out.   So therefore, when we're making a  decision thinking about a problem,   we're going to start with what's best for the  customer and then come backward from there.   Then in that coming backward process, we're going  to have to figure out, well, to do that, gee, I'm   going to have to solve this engineering problem,  or I'm going to have to figure out how to make   this thing cost less or make this thing faster or  solve one or more problems. That's the backwards,   that informs what's the work you have to do to  then create this new solution for customers.   Awesome. So just to summarize, you start with  what are the customer's needs and problems,   and I think a big part of Amazon's approach is  what are the lasting problems they'll always have,   which is I think it's lower prices, faster  shipping and all those things, and then think   with no constraints. When you work with companies  to implement this idea of working backwards,   is it always what is the customer problem and  need versus revenue or growth or something like   that? Or is there other examples of where you work  backwards from at different sorts of companies?   Well, the working backwards part is  strictly about the customer's needs. Yeah,   we don't want to work backwards from revenue. I  guess we didn't really use this term for sort of   other things like cost structure. Cost structure  was actually a part of working backwards from the   customer that if we had a low cost structure,  we could afford to give customers lower prices,   therefore let's figure out how to have a low cost  structure. Because in itself, driving down costs,   doing things more efficiently doesn't inherently  benefit customers because you could just choose   to take more profit. It only does if you decide  that in doing so I'm going to lower my prices   to customers or provide some other benefit. So no,  we used it in this method of I'm starting from the   customer, and then very specifically, we used it  in this method of new products and features that   I'm going to go build on behalf of customers. Awesome. This episode is brought to you by Wix   Studio. Your agency has just landed a dream client  and you already have big ideas for the website,   but do you have the tools to bring your ambitious  vision to life? Let me tell you about Wix Studio,   the new platform that lets agencies  deliver exceptional client sites with   maximum efficiency. How? First, let's talk about  advanced design capabilities. With Wix Studio,   you can build unique layouts with  a revolutionary grid experience and   watch as elements scale proportionally by default.  No-code animations add sparks of delight while   adding custom CSS gives total design control. Bring ambitious client projects to life with   any industry with a fully integrated suite of  business solutions from eCommerce to events   bookings and more, and extend the capabilities  even further with hundreds of APIs and   integrations. You know what else? The workflows  just make sense. There's the built-in AI tools,   the on canvass collaborating, a centralized  workspace, the reuse of assets across sites,   the seamless client handover, and that's not all.  Find out more at wix.com/studio. Okay. So then   when you go work with a company to implement  this idea of working backwards, what are the   very tactical things that you do to help them  here? I know PR/FAQ is a part of that, so let's   chat about how to actually implement that. What  are the steps to shift to working backwards?   Yeah. So the first shift is to take this, so  that's just a concept, right? Working backwards.   Well, how do I turn that concept into a scalable,  repeatable process? That's exactly where Jeff's   mind went. Eventually, without getting into the  origin story, we came up with this process called   the PR/FAQ process. So what it means is that  whenever we're devising a new product or feature,   we're going to start by writing a press release  describing the feature and describing it in a way   that speaks to the customer and to some degree  the external press and world where the idea is,   in my description of this, it better  jump off the page of something like, wow,   as a customer I will really need this. So what I work first is to say, okay,   for your product development process, let's start  by using this method as the method to decide what   am I going to go build? And oh, by the way,  to use it as a method to sort between a lot   of different choices of what you might build. In  summary, the way that process works is that PR,   you're going to describe very carefully and  clearly who's the customer, what's their problem,   and what's the solution that you're planning  to build. That sounds really simple and easy,   but it's actually very hard to do that well.  to crisply and clearly define those. The first   two things are the things that are hardest to  define, like who's the customer? Like anyone   says, "All restaurants are my customer." Okay, well, that's a mistake. Now, I mean,   which kinds of restaurants are your customers? In  what kinds of cities? In what kinds of formats,   et cetera, et cetera? Then what is the specific  problem you were solving? Ideally, you would some   way have quantify that problem or there's some  data or customer insights that have led you to   understand that problem, to know that it is a  meaningful and big problem. Ideally a problem   that people would pay money if you could solve  that problem for, because you can just look at the   economics of that problem, and if instead they use  your solution, this would be beneficial to them.   So I work to have them first implement this  PR/FAQ process is the first step. Then the   next step really is to go from there to say,  "Okay, writing PR/FAQs is one thing. Well,   how do I actually use them? How do we actually  develop them?" Because there's this iterative   nature to writing PR/FAQs where it's sort of a  concentric circle review. You start off small   with one author and with low fidelity writing  these things, and then you start to share them   with a small group and get feedback and improve  it, a wider group, get feedback and improve it,   and onward and onward until, depending on the size  and scale of your company, you get up to the CEO   as a way to strengthen, improve and really codify  this idea and determine whether it's a great idea   or not. So I help them understand how does that  work? How do you do this iterative process? Then   once you've done that, then what do I do with  these PR/FAQs once I've got them? How do I then   think about that with respect to my roadmap? Awesome. Okay. That was an awesome overview. I'm   going to fire off a couple of questions around the  first part. Do you still suggest people do it as a   press release? It feels like press releases aren't  a thing anymore. Do you ever suggest people do it   as a tweet or as TikTok video or a blog post? Good question. So the first thing is it's not   a real press release, okay? We could change the  nature of it, and if instead we wanted to call it   the customer problem solution statement, right? We  could just change it to that because there really   are three money paragraphs in this. First of all.  Yeah, it's not meant to be a real press release,   so don't use the language you would use if  you were sending an actual press release.   This is like an internal document.  Okay. So that's the first thing.   The second thing is the heart of it really is  that first paragraph, it's a short description,   that second paragraph, that's the problem  statement, and that third paragraph, that's   solution statement. If you wanted to ditch the  rest of it and the artifacts of the press release,   you could. I think there are other benefits to  it, like the headline, is this headline long   and drawn out and I can't even tell what the heck  this thing is from reading this headline? If you   used a tweet that wouldn't work very well. The date is also a meaningful thing when you   write the press release. The date is meant to be  a hypothetical timing on which you're envisioning   launching this thing which tells the reader  something. Are you thinking that this is something   that's so simple and easy, we're going to launch  it next month or so complex that we're going to   launch it in a year from now. So there are some  other directional cues within it. Like I said,   with everything, these are tools that people  can use and I'm sure that companies will find   other ways to improve upon these tools, but if  you don't use those parts of them correctly,   you're kind of missing out on what's the main  benefit that your getting out of this.   Do you try to write it in a way that would be  announced, like a press release feel? Or is it   mostly just who is the customer? Do you try  to pitch it as a part of this experience?   So you try to write it in that way, but the one  thing is you don't want to use hyperbole. It   would be very factual with numbers, data rich  document too. So again, not like a real press   release. A lot of internal confidential  data would be in this press release.   Got it. So it's a   tool that has a very specific use to it. Is there a template that we can point people to   in the show notes to help them craft this? I think  there's a version in your book maybe, but is there   some online that we could point people to? Yeah, so we have a website related to the book,   which is .workingbackwards.com,  and there's a resources section   within there and you'll find a template. Amazing. Okay. Then the concentric circle   piece. So the idea there is basically  get feedback from an increasingly   larger swath of the company and it sounds like a  big part of that is also get buy-in as you go-   ... swath of the company. And it  sounds like a big part of that is   also get buy-in as you go along the way. Yes and no. So first of all, there are some   things where you may write it and you, the author,  if we were in the old world, would take the piece   of paper, crumple up and throw in the trash can,  which is, in your own, you've realized, "Now that   I put this down on paper and read it, this is not  actually that good of an idea. I'm going to try   something else." By the same , you may then  have written one you think is a pretty good idea,   and you show up here or your manager and they  give you feedback that makes you want to then   ball it up and throw it into the trash can. So part of this concentric circle thing is not   just that everyone you write lives on and gets all  the way to the CEO. There are no stats in this,   but let's just say in some imaginary world  where, yes, all these things... You're a   product manager and you've got a director of  product management you report to who reports   to some senior vice president of division who  reports to a CEO. Well, if you truly run this   out and you write 100 PRFAQs in a year, maybe 20  of those make it their way to the CEO. The point   is not every single one of them is destined  to go that far. The numbers get narrower. And   this leads me down to the concept of what you're  really trying to create is a product funnel, not   a product tunnel. And with a funnel, meaning lots  of things at the top, fewer things at the bottom.   The tunnel means that everything that comes  in is also going to come out the other side.   And the problem with that method is that it  means you're not actually having a method   of consideration and comparing it against other  things that you might build or how you deploy what   are, frankly, most companies, your most precious  resources, which is your engineering team,   you should be looking at various choices.  You should think of yourself honestly as a   venture capitalist. They don't fund every company  that they meet with. They actually fund a very,   very low percentage of them. And at Amazon, we had  lots and lots of PRFAQs that were a great idea,   but we didn't ship them because we had  other ones that were just a better idea,   which had a bigger potential impact. So you want  that. You want to create this corpus of ideas that   are well-thought-out and select the best ones. It feels like a lot of these processes are   basically just ways to stop stupid shit from  happening. I think the narrative is a good   example where you have to expose your thinking  deeply. This is a great example of that.   Yeah. And it's also, I would say, an example  of where this is a process to prevent the   other process, which is the product development  process, from becoming the thing where you just   get locked in on, "What are we doing in this  sprint, what are we trying to get done," and   focused on shipping stuff. What I recommend is you  try to break that into two different processes.   One is the process of deciding what you should  go build, and that's what the PRFAQ is designed   for. And then once you've decided that, then,  yes, by all means, use all that good thinking,   freight, "Now how can I ship it efficiently  and effectively with few to no bugs?"   I was just reading this Harvard Business Review  article, I think that's called the thinking to   doing gap, where a lot of companies just spend a  lot of time talking about ideas and solutions and   not actually doing anything. And so I'm curious  how you try to avoid that at Amazon considering   there's this period of just like, "Let's  explore, explore, explore, and we're fine."   There's a couple ways, and of course I'm  somewhat having to imagine what are the   problems in such companies where that's going on.  So one such version of this problem is what I'd   call the-big-idea-that's-not-fleshed-out problem.  So I'm sure that every single person listening to   this podcast has either themselves done this  or have witnessed others in their company who   come up with a concept of like, "Oh, I think if we  built this, boy, that would really solve things or   that would really work well or that would really  grow things." And it may sound good to everyone,   it may sound good to you, to everyone, and then  maybe you start then working on building it. But   the reality is that actually once you've spent  some time looking at that idea more deeply, you   then start to identify several roadblocks or maybe  a fatal flaw with this idea. And in fact, no,   you shouldn't waste any of your time going into  building that thing because it has a fatal flaw.   So one problem is that companies get stuck,  I think, where they never actually go do that   documentation. And so it's a debate and discussion  about concepts that aren't really well fleshed   out. And so people's ability to actually evaluate  them in any realistic way is they don't have a   good way. And so in those situations, what gets  done is probably more of a function of politics   or will or a culture of completely top-down. I  think the other way is where they're debating   and discussing things that they just don't have  good methods where then they can take things,   and then go build them, meaning they probably  don't have the right org structure or processes   in place to then go take the good idea,  assign it to someone who will own it,   go look at it. And after they have owned it  and gone and look at it, if it works, then   they and their team can go actually build it. What I always found as I became more senior in   the company and my role became bigger and  bigger is that when something came up,   some idea that didn't neatly fit within my org  structure, I couldn't necessarily delegate it to   someone that this... There were only two things  I could possibly do, which is just set it aside   altogether because otherwise it'd just be a  real distraction to people or I had to decide   this was a compelling enough idea that we were  going to take a resource, could be one person,   could be a whole team, depending on the idea,  and I'm going to have to assign that resource   to actually go look at this and work at  this. Otherwise, it will never happen.   I've been through those many times. Okay. So  there's two more concepts I want to try to touch   on before we wrap up. The next one is the idea  of input and output metrics. This is something   that at Airbnb, we super implemented, it became  a very defective way of thinking. And actually   there's a lot of Amazonians that ended up at  Airbnb, a lot of leadership. So there's a lot   of this stuff that we ended up doing like the  memos. And so on the input and output metrics,   could you just describe what that is and  why that's so important, why people think   about metrics in the wrong way often? Yeah, so the origin of this one really was,   again, in our early years at Amazon, '99, 2000,  2001, we were a public company then, we were   growing. But then growth started to... It wasn't  just all up into the right and like, "Woo-hoo."   Every company's going to hit a wall eventually,  and it's not going to be... If you're so lucky   to even been at a company where it's just going  up into the right with no gravity, good for you,   because million people never experienced that. What most people experience is the reality   is that there's a lot of gravity pulling against  your revenue numbers and you've put a plan out   there and you wanted to grow 15% or 20% or 75%  or whatever it was, and now you look like you're   not going to hit that number this quarter. And  so what ensues then is, "We're not going to hit   our number. What should we do about that to hit  our number?" And this often happens with, well,   there's a month or a month and a half left in  the quarter, and then we would run around like   chickens with our heads cut off and come up with  a bunch of ideas that tended to be promotional   in nature and tended to be price reduction in  nature, or we'll send this extra email or extra   ad or whatever it might be- Another Prime Day.   Right. And the reality is we did that, we went  through that enough times, several quarters,   and we started to realize, "Huh, these fire  drills don't really work." We didn't really get   meaningful progress against the number with these  last-minute things we decided to go do. And oh,   by the way, they were a big distraction. If they  did work at all, they pulled revenue that might've   just gotten in the next month or next quarter  into this one. So it wasn't really a zero-sum game   there. And we realized we're not really actually  working on things that matter to customers that   are going to move the needle over the long term. And this is about the same time when Jeff and the   S-Team were reading the book, Good to Great.  And you have to ask Jeff what it is, but if   you ask me, I think that this was the single most  influential and effective management book for our   company because what it caused Jeff to do, and I  won't describe what... Most of you probably know   what it is, if you don't know what it is, go read  Good to Great. It is, in my opinion, the best,   most important management book you'll ever read.  Because what it did is to help us codify our   growth flywheel, meaning what are the inputs that  if we improve these things, which in our case,   was how do we have broad selection? How do  we have a great customer experience or great   customers experiences in retail? Things like  how easy was it to find what you wanted to buy,   how easy was it to buy it, and how fast  did it get to you. Were the prices low? Do   we have lots of merchants on our platform?  And by the way, could we drive out costs?   So we identified these things on our flywheel.  And this identification of these things was such   a critical moment for the company because then  it realized, "Okay. Well, what we need to do is   spend our time focusing on how do I measure each  one of those things, and then how do I improve   each one of those things?" So it shifted our focus  away from this short-term thinking of pushing the   revenue number up to this longer-term thinking  that if we just improve these things, whether   it's... There's no day that people will wake up  10 years, 20 years, 30 years from now and say,   "All else equal, I'd rather shop at a store with  fewer items than more items or a store with higher   prices than low prices or a store where things get  to me more slowly versus more quickly." So if we   can just improve these things, this is our path to  winning. So those were all inputs to the customer   experience. And so we then figured out ways to  measure them creating a set of input metrics.   And so then when we would develop our operating  plans and review our business each week and set   our goals, we were hyperfocused on those inputs  and the input metrics. As a simple example,   there was one tool that Jeff and the leadership  team, the S-Team, used called S-Team goals, which   are effectively a list of what they would harvest  would be like, "Here are the most important goals   for the company that I've harvested from all  of our operating plans." And I can't remember   exactly what year, something around 2007, 2008,  they looked at that list, which is about 500   items long by the way, and they counted it up.  And of that list, only 10 of them actually had a   financial metric in it, like revenue or free cash  flow or gross profit. These other things we're   generally speaking, all... One of those inputs,  like I mentioned to you about low prices, and   selection, and speed of the customer experience. So, yes, the point was, again, it's this other   article... So we took it as an article of faith  that if we can just improve these inputs, the   outputs will take care of themselves. The inputs  are the things that drive the outputs, which are   revenue, customer activity, free cashflow. And  so one of Amazon's... It's not really a secret,   but one of Amazon's great strengths is [inaudible  01:00:28] focus on those things and make just   continuous process, continuous improvement on  each one of them and measure them rigorously.   The flywheel, you reminded me. It feels like  that's another concept Amazon proliferated   through all of companies is everyone's  trying to create their own little flywheel,   and I imagine everyone has that image of the  Amazon flywheel in their head with a little   orange circle in the center and the black arrows.  On the topic of input metrics, just briefly,   what is an example of a good input metric? Because  I imagine people that are listening are like, "Oh,   shit. I got to think about my metrics  as input and output now." What's a   sign that's a good input metric? A sign that's a good input metric is,   first of all, map your end-to-end customer  experience. I never worked at Airbnb, but,   okay, step one is that they clicked on some ad  somewhere and showed up in the website or the   app. Now you're in the app. Now you're looking  at this first screen. Well, the first thing,   what they're doing is they're browsing and/or  they're searching. Okay. How are we measuring   the speed, quality, and ease of that browsing and  searching? Now they've got onto a detail page for   an individual property. How are we measuring the  speed, ease, and quality of the different actions   they may take like reserve... Forgive me if I get  any of my terminology wrong. I'm not an Airbnb-   You are, but it doesn't  matter. It's close enough.   So then you've reserved. Now you have interactions  with a property owner. How do I measure the   quality of those? How many messages go back  and forth? Is a lot of messages a good thing?   Is that a bad thing? At first, you may not know  the answer to that question. Same thi