Collaborate on Invoice Tracking for Engineering with Ease Using airSlate SignNow

Invoice tracking for Engineering

[Music] hey everybody maria how are you i'm good how are you phil great i think we've got our our av issues all squared away it sounds like we can see and hear each other very well people are clapping in the chat so that's nice um yeah i welcome to the thanks to thanks mustafa thank you to the to the plato community and hello everybody thanks for for taking some time out to uh to chat with us i think marie and i are um you know we were excited to to get together for this this is a topic that's near and dear to our hearts and and consume some chunk of our professional attention um and so uh getting to share it um with with you and the and to field questions and and kind of talk freely about it is a is a fun way to spend a little bit of our time so um with that i'm going to uh ask maria to introduce herself and i'll bring up a few slides that we have uh for the background here thank you phil and uh i've been watching the hellos uh from all parts of the world go through on the chat as we were waiting here in the what's called the green room in this tool that we're using and so amazing to see people literally from all around the world i saw dubai i saw and an old colleague shout out to martina who knows who i am we worked in our first jobs together a long time um so uh you're gonna bring up the slide right i am i can get started on my intro so yeah yeah yeah i get started i uh work on my way through there maria vp of engineering at toast currently uh my career has taken me from where i started in electrical engineering in ireland uh soon thereafter to the boston area multiple companies currently at toast and toast uh serves the restaurant industry with lots of butchers just a little bit a little bit further along but my experience with engineering kpis has extended probably through my last four or five jobs and i've had more senior leadership roles and what i found in companies my first vp job was a company of 25 employees uh toast is much larger than that um but there are some things that are consistent in engineering kpis across all of the companies that i've worked with and there's something that is things that are different for example the industry that i'm working with or if it's sas versus on-premise or the for selling to consumers versus selling to bnb enterprises and so i have i think a bunch of different experiences relating to metrics that an engineering leader will track and uh welcome questions on the chat because that will help us to focus in on the areas and metrics that all of you are most interested in awesome thank you maria and so i'm phil i'm one of the co-founders of jellyfish we make an engineering management platform that actually helps uh helps quantify this right helps you understand by by actually measuring the work that you're doing in jira and in git and other systems and and making sense of it in a way that that the rest of the company can understand and um you know professionally i've been doing this for for 20 years and this project i think for all of us who started the company is uh you know a labor of love out of like the operational experience and the struggles that we've had over the decades you know trying to do this well i think it's always been a a challenge to both quantify what's going on in engineering and also to talk about it in a way that senior leadership understands so i think we've we've brought that perspective to to what we've built here and we've had a ton of fun over the last four years working with hundreds of organizations and thousands of teams and lots of great people like maria i'm psyched that she's here to to do the webinar with us um i think with that like we can we can maybe get into some of the topics and then as the as the questions um pile up then we'll uh we'll go to q a so so the the first slide that that i have is just a nod to we've been producing this like state of engineering report we've got a new one coming in the in the not too distant future and so this is a little bit of a preview of of data that we've been collecting for it and it's just a a way to reinforce what we've been learning for years here which is that um you know as trite as it is like you know what gets measured you know gets improved and uh you know at jellyfish we have a particular point of view on it which is that you've got more leverage and it's more important to first measure you know what you're working on and where you're focused um and then once you're once you're doing that once you're pointed in the right direction then you can measure you know then it becomes more relevant you know how how fast are we going how efficient are we how quick are we how collaborative things like that and so one of the findings that you see here is that um you know we get a lot of good kind of before and after data with people and you know consistently over and over and over again once you start once you start measuring once you've got a you know better framework in place for this you're able to spend more time working on the projects you care about innovation projects roadmap projects that's that that's at 25 and then kind of the maybe the unexpected corollary is that um you actually end up spending a little bit less time on on you know thrashing through all the support work it's not like bugs don't need to be fixed but i think you know once you've got um full view to everything you care about you become a little bit more efficient i think that also kind of plays into some of the productivity metrics you see at the at the bottom or like the operational metrics where you know this isn't the target the target is not to you know have more pull requests or i mean sometimes the target is actually to decrease cycle time that is actually is a good goal that we see lots of people care about and focus on but i think um just the mere act of being mindful and having a tool of like okay this is what we're working on this is what we're trying to get done this is how we can see it uh time and time again you know it pays dividends like you know down the stack of kpis and so it was nice to to get an update of that data and kind of see it across so many companies i think with that i wanted to um you know let maria like you you may if you can tell us a little bit about toast because i think we've you know we've been working together for a while now and you guys have a uh a pretty interesting story you know both as a company and as an engineering organization and it'd be great to share it that's right thank you yes toast has been working uh with jellyfish from prior to when i joined host i've been there two years and toast was well established when i joined the organization uh we are a group of 250 engineers uh the little stars here indicate that we have offices in boston which is headquarters as well as dublin huge innovation center there and chicago which in the chicago office manages our employee uh product we have uh one platform three different user groups so the platform has in fact a bunch of discrete components within it and products within it to to serve the three user groups who are the guests of the restaurant so some of the products that we build are directly guest facing online ordering order pay at the table for example obviously much more important these days in colgan times and we have to serve the employees of the restaurant um trying to operate smoothly and we want to serve the management of the restaurant who is running a small business and needs data and reporting and employee management now we now uh are installed and operating tens of thousands of restaurants in the united states and the context that i give here is not just about toast but to give the sense that this is a fairly substantial organization um and probably the most data driven company that i've worked for and so i've actually enjoyed engaging more with data and orbits engineering metrics including the jellyfish metrics while i've been working at toast and can you tell us a little bit about your like your software stack i mean aside from jellyfish like what do you guys use to build and manage manage your software process yeah so uh we're cloud native so that's an aws java react front end stack in that middle picture middle column and you can see a kitchen display so toast also has handhelds that are used in the restaurant as well as kitchen displays as well as terminals and restaurants and those are an android app and so a bunch of android development in kotlin and one of our technical challenges is the communication between the fleet like tens of thousands of of devices and restaurants with the back end and so again to the extent of data uh we have a data platform um which contains the orders data as well as the employee data and the guest date and restaurant but the data of all of the orders that's flowing through the platform uh so again multiple different engineering teams serving the products and the platform that services from customers jellyfish integrates tightly by the way with jira and with github which we use as part of our foundation awesome um and so i think the next slide is like talking about a little bit how how specifically you guys think about metrics not just um you know kind of you know at the core engineering level but maybe across the entire engineering organization that's right that's right and you said something earlier i felt around when you measure it you can impact it making it transparent means you you take action on it and that is certainly a toast so um you know i think there's a few fundamental truths death and texts people talk about but another one is that if you're an engineering leader you are going to be asked by your management whether it be the ceo or the leadership team or even the board like is your team productive are you working fast enough you have x and for me the x has been from ten to hundreds of engineers what the heck are they all doing are you productive enough um and uh i've received this question appropriately my leaders are trying to run an efficient business um and that i have put on the bottom right of this particular slide because i think that is essential um information that as an engineering leader i need to have and i need to track but i always want to bring the conversation broader than that as well when i'm talking about engineering metrics uh to me one of the more important places that i need to make sure i'm optimizing for is that we're working on the right things this of course is a partnership it's a product management engineering leadership user experience partnership and at toast we have what we call product health scorecards which look at um what are the kpis that we're working towards for particular products around sales metrics activation adoption impact for customers as well as the customer sentiment as well as the product quality are we meeting the slas that we have are we uh what is the support traffic for the different parts of our product so so when i look at engineering kpis in aggregate this is absolutely a collaboration with product um but i but i like to make sure that i include product oriented and impact oriented at kpis right moving to the right and this is where the integration with jira and jellyfish and other systems comes in is it's helpful to have a holistic point of view for planning and for reporting and for managing teams around our product investment and our success in delivering to our road map so we use roadmap allocation at toast we talk about multiple categories four categories of investment from things we do to run the business keep our current customers having a great experience things we do to scale the business to grow the business and to maintain or improve the profit of profitability and so we like um through jellyfish to end and jira to roll up what we anticipate will be our road medication across those puppets as well as what actually is which is when jellyfish comes in and to see what where did our team spend their time based on the tagging that is rolled up through these systems and of course we track how we're doing with our initiatives are we delivering as expected to the roadmap and the initiatives that we're setting out right so some of the stuff is um in fact i'll even go to the next slide as i ask this but like some of this stuff is um you know pretty pretty traditional and pretty well baked like we all have meetings where we talk about the status of key initiatives and when they're going to get done and there's you know usually some reporting framework where that gets circulated to senior stakeholders i think when you get and maybe i'll even keep it at this level before we get into the to the to the next slide um maybe talk about how for some of these other things like who engages with these kpis like and how do you like incorporate them into you know the different organizational or operational workflows at toast because i think you know there's one version of this that i've seen where people compile these stats and they go into some you know giant you know operations template but nobody really uses them for anything yeah and then there's and i know that that's not what happens at toast and there's probably a longer story than we have time to tell but maybe just a few highlights of how you guys actually put these things to work would be really interesting to people and that i think is another fundamental truth that you're getting out there phil which is uh it is it is not helpful to gather metrics uh that are not put into practice for our for archiving purposes in a database of metrics right and it's also um counterproductive to gather metrics and share them with one piece of the organization but not transparently open them up to the rest of the organization um and so if i look at these metrics here the product health um very strong collaboration across product engineering and customer success customer success are our team that's on the front line talking with customers and so they are um very uh we want this as tight a communication as possible between them they're hearing the goods the bads and the alias of our products our customers experience of the product with the product and engineering organization so um there are many pieces to this it's sort of an accumulation uh conglomeration of multiple metrics and so the audiences so the contributors are many because the product quality slos tend to come from the engineering part of the organization the adoption activation is from our product and customer care the customer sentiment is coming from customer repair but the audience is senior leadership looking for trends in all of this as well as the product engineering leaders ourselves to see well are we making our goals for uh the kpis and the adoption and activation goals that we have right like this is this is that on that ladder point like this is stuff that you discuss with with your staff with your direct reports yes and then they carry you know carry forward and discuss it with with their team leads and so forth we have quarterly planning and many levels of the organization the r d which is product engineering organization involved in the planning and so um at that cadence minimally we talk about okay what are the goals the kpis that we're setting for each area of the product for that next quarter and then in the area of product investment and delivery on the right hand side this is on a cadence of at least weekly for all team leads and managers in r d got it got it there's uh i'm gonna i'm gonna jump to one of the questions from the chat because i think it's something that you and i have talked about like not related to this webinar before um and it's one that's gonna continue to get upvotes i predict just based on you know doing this uh a fair amount um the question is how do you strike the right balance between getting engineers right like so you know obviously you have to run this through like different levels of leadership but at the end of the day there's the people that are writing the code and moving the jira tickets around how do you strike the right balance between getting them to like to care about the metrics but not to in in the in the words of the of the asker here like not make them an except an obsession and succumb to gaming the metrics for short-term wins like i feel like that's a that's if there's a frequently asked questions document here like that that is going to go up at the top so i love this question yeah and in this case i'll focus on the metrics that would show up under productivity so for example cycle time release cadence those are the ones that the engineers have the most like impact right like those are the ones that are kind of like you know up to them in a way that like the allocation stuff it's a little harder for them to be too invested in it or at least two you know they're indirectly affecting it not like anyway keep going yeah i totally agree and what i think is critical here is that there's the ability to zoom in and zoom out of that level of granularity so when i uh talk about productivity metrics to a senior audience i'll zoom out this is about trends and aggregates and when an individual's scrum team maybe a team of five is looking at their metrics they'll be zoomed right in and so what i want to do is take away any sentence of it being punitive or any semblance of it being um sort of micro managing and that ability to show high-level trends and high-level maybe blockers and challenges that the team is facing and but make it actionable and relevant to the team members that are doing the work i think that's the trick might be the word or the perspective that we need because it's not helpful to show individual and bits and bytes to sort of a broad senior audience and that would of course feel threatening it should that's not a super healthy environment set up but at the same time if it's all high level and all trending it's not very actionable for the teams that are working in the code base and working on the recycles and so that's the balance that i try to strike which is right level of the information for each audience although the transparency means that we all understand what the big picture is and where we're headed right that's that's a that's a great answer and i think there's one other thing like if you were asking me that question you know what what do we see there's you know there's a thing that i think we found over and over again and that we try to try to reinforce and i think toast actually does a great job of this which is instead of putting because actually if i skip to the second question the second question on this list is what are the top three metrics to look at and that's a good question like what should the three be um sometimes i'll get the question like what is the one i kind of want one thing and so the more narrow you make it the more likely people are become obsessed with them and or you know god forbid actually try to game them a little bit and i think certainly in toast's case you guys look at this nice cross-section of things um and you do them as like you know a secondary thing not the primary thing and so by because of those two things right if if all we cared about was um deployment rate or you know number of pull requests or story point velocity if you only cared about like one of those things that then you know the obsession obviously kind of follows and i think by by having a few like yeah you should diet yeah you should exercise like there's a few different things you should care about but like not not any one of them is put on the pedestal as like the thing that is indicative of of engineering performance um i feel like that goes a long way to keeping a healthy healthy outlook from the engineering team that's a great point for the other thing that i'll add to that you prompted me was that the cadence and the expectations are different team to team but most definitely part of the product and a lot of that on the stage and maturity of the product or the technical staff that you're working on so for toast for example um we have microservices as well as as on the core web platform application and that releases frequently all the time there are microservices daily released by different teams uh compare that or contrast that to the point of sale application which is a large android application uh we're very very careful in how we release that we have preview pools and we have scheduled releases that are multiple weeks apart and we carefully elevate to the customer base making sure that the first cohort has a great experience before we elevate further and so it's impossible to compare cycle times of release and sometimes even stories but for sure releases across those kinds of teams and so i think that we will always encourage comparing with yourself or comparing with your close working peer groups uh versus comparing in absolute numbers with you know another team that's not a related team for those for those kind of metrics which you know i think of as operationally important for how teams and and product lines are running um but maybe not uh like cross-functional executive friendly you know for lack of better phrasing like it sounds like it sounds like you you you do share some of that stuff like i mean i think toast everything i can tell about it you guys are super transparent as much as possible so you'll the question is going to be if you share that information cross-functionally does anybody care like is anybody looking at like the deployment frequency or the you know the story point whatever the metrics are do cross-functional leaders care about that stuff how do you how do you deal with the fact that like you might have metrics on display that people don't understand yeah the jellyfish metrics are not broadly shared like the customer success or marketing and sales team and if somebody would have access central could give them access but it's in it's not in their day-to-day um but when i share that sort of metric and those sorts of metrics with my leadership team it's often to advocate for resources change a process new tooling investment and test automation um and so it's it's often shared in order to take action uh on what i'm seeing now it's very much shared within the technical teams themselves and in that case um what we want them to do and what they do do is sort of look at let's say it's a cycle time and then break that down and say where are we getting stuck within that cycle time how how does our team do with uh story reviews and pr commenting for example and trying to remedy those so i think that there are different levels of visibility and the productivity is shared in certain places um with a non-engineering audience and but in that case it's more often that it helps us make the investments that we want to make that's a that's a great answer and it like um you know there's there's always been this relevant analogy i think you know from when we started the company throughout which is we look at how other departments run you know in terms of metrics and transparency and sharing those metrics and do other you know parts of the organization even get it and so like you know sales is is the is the part of the org that um you know usually historically has been most forward on you know producing you know we've all used salesforce for a couple decades now and like lots of tools on top of it and so you know i can't count the number of times i've sat in a in a meeting where like sure we're reporting on pipeline or activity or some things that you know cross-functionally people aren't like dug in on but when it's time to make a decision or raise attention or you know kind of call people to action having that data where it's like hey look we've been looking at pipeline but now it just went up that means we need to hire some people or now it just went down that means we need to do some marketing if you didn't have if i was in a different part of the org and i saw that data for the very first time it would be really really hard to rally around it and so i think one of the things that we're finding is is with companies putting this kind of data forward where like yeah they have it they're looking at it so some of it's like a little bit of a show your work like cya-ish thing but it but without doing that you can't actually rally people to take action on it because the you know they'll be just blindsided but like what does this number even mean um rather than like oh yeah this is the thing we've been talking about for a long time but now it actually matters because we need to we need to do something about it um so that definitely resonates i totally agree that often the in-team conversations are around how do we improve what we're going next how do we make this better and then the out of team is about the trends and what do we see and how is it shifting and if we make an investment here's one that i really do care about if we make an investment for example we want to make it easier quicker more out of the box to create and deploy microservice and we invest in that for a quarter well i want to have the metrics so when i come back at the end of the quarter it's not just me it's the team coming back at the end of the quarter to say we invested in this tooling or templating or devops enablement and see these are the teams that engage in the pilot and here's what we can see for their ability to deploy more frequently again having that foundation of measurement and having that vocabulary that we're able to um show that engineering is looking for roi and investments the same ways we would expect any other team to look for roi and on their investments is super helpful for the for the executive audience any audience but sort of so it's less about the specific of what you're measuring it's more about being able to tell that story about this is getting in the way of excellence so this is getting in the way of the impact of my team and i will show you how it changes right as we progress along the path that we want to right right well here i'm going to i'm going to go to the next slide because i think it's it's relevant to continue this and it's also a little bit related to some of the other questions people are asking um i mean some of these are like right some of the the specifics to make it real yeah did um well like so so there's there's a you know there are a bunch of questions here and we have another slide on on metrics too but um the one i'll grab uh from the list here is um i'm curious whether you measure and share metrics around things like code quality right like things that aren't directly valuable to the business at all but uh you know are do affect the team's ability to deliver values sustainably in the future um so either either just specifically on code quality or other things maria that you think are kind of in that same bucket how do you handle them yeah um yeah as an engineering leader this one is critically important to me um because every team has to find that balance of innovation go fast release to market and be sustainable build something that has high quality and that's architected well to scale and grow and and we have that exact same set of conversations and trade-offs um at toast possibly with different baselines depending on the maturity of the product and the number of customers that are depending on us for that area of the platform and so why i uh so a few things that we do one i previously just touched on investment allocation and so how much do we expect to be investing in and i use the word run the business but within that we we call um the customer reported bugs that we need to respond to so that our customer sentiment and our nps from customers is high we also have a category that we mark in jira as the scale category and a lot of times our technical debt will show up as a scale initiative or a scale set of jira stories and ethics and for scale i mean um it was architected which worked just great when toast had five thousand ten thousand customers but now we're tens of thousands of customers and we need to re-implement uh a piece of the technology to scale in that way uh and so those are two of the um those are those are two categories of investment areas that we track what we expect to spend and what we did spend quarter by quarter and it's going to be different for different areas of products but it enables us to see how much you'll be spending on scale and tech debt for example and then totally in addition to that um in a separate sort of cadence we're looking at quality metrics in particular our uptime our fair customer interactions with the product as well as customer reported bugs support cases that come in for um different features that are that are new or that are released and in use in products and so that is fully visible [Music] i think very important that customer care and r d product and engineering has the same perspective on how we think of the quality of a product and ultimately the quality department is the quality of customers experience at that product there's technical debt that we sort of have to do in a different way and hopefully it's not that visible to our customers but the customer experience quality is in that product scorecard very visible shared across the company and i think important that we as a as a strategy team have the same perspective on how we're doing that's that's a good answer and that that like certainly it covers the i think covers the like executive half of the question i think and if i'm if i'm guessing though like there might also be a you know closer to frontline manager you know i've got a team of engineers i'm trying to make sure that they actually write code well and that they're you know properly trained and that we're you know we're hiring even the right kind of people in the right geographies and so on do you guys do anything to um and this is stuff that like you know just being candid like is a is you know a bit out of the the realm of like where jellyfish like spends most of our time i think we certainly talk to people about it but we're not investing heavily on trying to like determine is you know engineer x writing good code versus writing you know not so good code um but do you do you guys have any kind of like top-down framework for that is that a manager by manager kind of thing like how do you how do you how do you deal with that question sure so we of course want every developer to write with best practices and high quality code i don't think we have a what i'll say a program or sort of a formal way of managing this we have multiple informal ways and code is called reviewed before it's um the pr is elevated and it's sent to customers and so um i would expect that if there is a particular maybe muted job developer that has many elements of feedback in the code reviews that that's noticed by the team lead we have a small teammate which is really a hands-on working with the team but in the leadership role taking the leadership role and i think that while it's not explicit in our process and i think that it would be implicit in the number of comments and feedback received in the code reviews i think the cycle time matters and so we certainly can see it's not something we go hunting for but we certainly can see if there's anomalies in the cycle time of any individual developer um in completing their stories completing their epics and that would be an indication of they do it and you know code of you finds issues or quality testing finds issues and so i don't think we have a system for that but it's certainly um i would hope that the normal process of writing code will help us identify yeah we see we see that too like like then probably that's also why we aren't you know endeavoring to like go so deep that we're trying to judge what is good code and what is bad code because if it's you know the the systems that are in place you know kind of work themselves out if code is taking a long time to to get merged like that's an indication that there's some something going on there and if and if um issues are taking a long time to to ship then that's you know a similar a similar thing like either have you know code review cycles or qa cycles that are that are just dragging things out so i do think that that cycle times are like the you know kind of the ray of sunshine that helps helps you uh you know see it like the managers can see it because they're in the thick of it and then if you're skip level from that you can start to see it in those in those indicators like we see that all the time that's a good answer i've sort of responded to your questions but i am interested and you have a much broader perspective and given that your customer base so what view is an ideal usage of cycle time quality and the what what jellyfish can serve up to a team and to his leadership um well i think i think that there's a there's a even a slide we have later about talking about metrics and you know which ones do you pick and there's you know but there was a question here about which are the three and cycle time is usually um you know one of them or maybe even two of them if you're looking at kind of like the you know the get level cycle times versus the jira level cycle times um and i think just general advice not not to ruin that slide you know i think it's fine that we're kind of off track of slides now but um you know general advice for picking metrics is that you should pick metrics that are relevant to your organization at the time and so that is like part of the broader perspective of you know working with so many people is that like some people it's like really important that they you know work their way up the devops maturity curve break their work up into smaller pieces you know build more automated testing and so their you know throughput and and cycle time metrics become more important there um than in an organization where they're just getting buried and sustaining engineering and support and and can't you know find time to to work on innovation and ship new products so you know depending what you know what to focus on you know is contextual and it depends on what's going on within your company and i think that can change um you know so like you know metrics may evolve year to year or you know a couple times a year they shouldn't evolve meeting to meeting right if you're in that in that mode where um you're just thrashing around trying to measure one thing and then you know completely you know measure a different thing the other time that's no good and i think like specifically to your question about cycle time right there's you know part of it is helping make sure that you know at get level it's making sure that the engineering team is working well together yeah um and getting things you know quickly through that workflow sometimes there's a devops component to it if like the you know the ci cd process is is a big part of like actually getting the merge done um and then separately the jira level cycle times are more um useful around are we you know plugging into the business the right way like do we know what we're supposed to be building are we able to actually get it out the door you know for running sprints and our cycle times are like well under the you know the length of a sprint that's good if we're running kanban like are we you know measuring things in days or you know you know single digit numbers of days rather than that longer um i think those are i think those are you know really important for people do you find uh that the engagement is at the team level or the team leadership level or at the organization leadership level i think that um you know part part of the trick or you know one of the the things that we've seen our most successful customers do yourselves included um with metrics is you know having some amount of top-down uh pool for for for certain metrics so like if you maria know that um you know there is a good reason to focus on cycle times and getting faster things either you know on the on the you know jira and with respect to product side or on the git with respect to how the team and the deployment process work side just by just by you know making that one of the top kpis and by you know talking to your senior staff about it and having them you know report on it is is um an important part like that that sets the tone for it but then like the real operational impact is like okay now that we know that we care about it right if you know if you tell me to exercise more like okay i got it and then the way that each of us end up doing that we you know we probably do it differently and it's not always um you know again this is super dependent on the culture of the company but it's it's it's usually not that's not necessarily you know orchestrated or like defined for everyone at the top like that can be you know that actually can be federated like group by group and leader by leader yeah yeah i actually love that approach and that recommendation or when when you say you see you see best and class organizations because i do think that metrics enables accountability to the metrics that we're driving towards with the reach team to do it their way and do it in a sequence and and with the collaborators and the participants that are going to be able to make a difference and they're a part of your organization so yeah i hear your point about the visibility but not the fine micromanagement of how awesome this i'm i'm enjoying there's a good answer and i'm enjoying how well the the q a and up voting process is working because there's like now the top question you can see it too it's like is exactly related to this like how have your kpis changed and evolved you know over time right because just like i was saying like they might change you know you know year to year or you know they shouldn't change meeting to meeting but they probably will evolve how have you evolved yeah that's a great question and and before coming into this webinar i thought about how they changed from company to company and it was primarily around the the type of customer that would probably be the generalization so is it a b2b in which case it's uh some of the enterprise customers care deeply about regressions because they're deployed based within their companies and uptime of course but at toast we're looking at a much much broader set of users consumers to guests in the restaurant to restaurant employees and so they're much more sort of the high volume b2c sas metrics that we care about um on the engineering side they're the same but on the kpis that we're driving to as a business they are quite different and but then to take this particular question as as a toast let's say that our organization has changed how our kpis changed and and i do have an example um i think as the company grew the number of independent but but collaborating pieces of the platform grew and so as an example two started as a point of sale company a company for restaurants and that means in the restaurant and being able to take orders from customers and kitchen the orders get back to the table and then check out at the end of the meal the software that supports all of that well over the last few years and in particular last year we released a bunch of guest facing software so for the guests of the restaurants to be able to do online ordering organize curbside pickup and order and pay at the table but it's at their driving versus the restaurant employees writing and what we have found is that the kpis that we had to rethink some of our kpis in particular it was no longer any good to have each team think about their individual kpis we had to chain these together so that we have to think about a customer interaction with the product which scanned some of the guest facing to the restaurant facing to the back end platform and data platform pieces and so that's an effort that we're actually engaging with one of our priorities this year is better together it's sort of become a bit of a buzzword but it really means that we have to step back and think holistically about all of the kpis that we're doing and these are more product related but certainly also engineering related from an architecture perspective so that we're thinking more broadly and measuring the things that matter in a more complex multi-product platform that's that's that's a great example i think i think broadly we'll like just to answer it kind of on behalf of like you know what we see at jellyfish i think there's there's a couple classes these one is like a scaling the organization you know thing like so as the you know in the beginning like throughput is high and cycle time is low and you know you're not you know maybe you're worried about quality like and handling support you know you need to make sure that you fix the things that are broken but you're not really you know spending the majority of your time on sustaining engineering so you're not worried about that so like as the as the organization scales it's you know putting kpis in place that help you measure that um and we also see ones that like will will evolve where the the pendulum will swing between like do we really know what we're working on and have good you know visibility and allocation to that and and if not well then the metrics are you know the kpis and the goals are much more focused on that and then once you know you feel like you've got a handle on that and kind of the process is working the alignment is good then you know there's more focus on let's let's improve our cycle times let's improve our devops metrics um you know let's work on you know the efficiency of the machine and and uh so sometimes i see you know also companies kind of like swing back and forth between those two things mainly just to you know put attention on the thing where they think there's you know the most opportunity to to to kind of where there's the most leverage right yeah another aspect of growth this organization and others experiences when you go from a single site to multi-sites and you have work that um and it requires collaboration across the sites but sometimes even sort of passes the boundary with the engineer engineers and multiple sites collaborating or quality or devops engineers and what is like collaborating and at that point it really is important to have visibility into the overall um how the system is working as a whole and so that a we can learn from best practices from different teams but b we really can troubleshoot and diagnose where there seems to be bottlenecks in the process they're not always as apparent that's when everybody is our routine some of this of course sounds moot when we're in a culprit work from homeworld and i think the visibility and the network still still helps yep 100 uh the you you can see to the there's a chat question here of how do you measure the the cooperation and collaboration part which again i think you know whether it's multi-site you know pre-covet or it's you know post covert we all need to make sure we're um you know collaborating the right way and and not just kind of in my attic on an island up here um yeah how do you how do you think of that yeah we put a lot of emphasis on this and completely outside of metrics we encourage um all of as you get more senior 3 4 5 for sure and that the team is contributing to the foundation of the the r d team as well as the products that we're producing so a simple example is being engaged in the interview process being bringing new talent and sort of contributing to the culture that's not measurable in this way within the systems that we're talking about here we do look at um again in a mix of other metrics how teams and individuals do with code reviews and this most often comes up in times of promotion uh when we make a case to promote somebody to a more senior engineering role we look at okay are they contributing well to delivering code reviews um are there code reviews we can look back if that's important to the context of the sort of feedback that was given and that and collaboration and onboarding of others in their team is certainly part of our vocabulary and it's nice that it's supported by data in this case that like this actually this is such a good point both for the this this topic and then kind of more broadly um you know what you're saying i think is super powerful which is that you know there are some metrics that help you give you some indications of how people are collaborating um but but the real work like the things that are you know most intentional and most important for for your organization around collaboration are like they are codified they are things that you're very intentional about but they're not things that show up into you know in a in jellyfish or any other dashboard i think i don't know if you have a dashboard to see you know who track which engineers are in interview you know part of the interview process and which ones are you know doing other culturally important contributions but like making sure that that stuff happens as people kind of work their way through the company is is is that i mean that's that's your primary answer right that's right that's right and one element of collaboration is collaborating across the different functions within an agile team so um the quality software devops and product and ux team members all have to collaborate and i think the collaboration shows up either positively or not so in the metrics although not explicitly called it yeah and so it's often conversation that starts because a team scene the team seems to be trending down in their cycle times or their throughput of stories in their background and often what's uncovered is well um the the requirements were not clear or we had to wait multiple days for somebody to respond to this api oriented question and so though they tend to come out in the deeper direction we find that we're trying to troubleshoot right so so okay so keeping track of the uh or like keeping your eyes and ears open as you go through the development process and the different ceremonies that you know brings some of this stuff to life right and i'm i'm asking maybe i'm guessing like you mentioned earlier in the in the session that you guys do employee nps score like the some of it does some that bubble up that way too i'm guessing yeah it totally goes that's a good point anything like that that that number itself doesn't like say like oh you're collaborating well you're not but like that's kind of the level of of data that you get we're like oh okay like this part of the org is functioning well together and you know here scores are dipping or people are raising concerns totally agree though it's certainly you'll see what we'll use the words team health show up and we happen to use culture ramp at toast sorry we'll see team health and then typically there's many questions on these surveys and we drill in and do people feel like they're well supported people feel like they're collect their peers are helping them be successful so yeah it shows up but i think you have to dig one layer deeper together right right there's no there's no easy way out you don't just program the robot to like grade everyone's collaborativeness and then and call it a day right like it this actually takes managers doing management yeah um i'm having fun going through the through the uh through the questions here um there the there's one how do you link kpis with okrs and do you use tools other than jira to track the kpis oh i wish this one was easy but it's not easy so we do have book cards and we put work in to form the kpis many times the kpis are not the sort of engineering productivity kpis that we've discussed here the kpis are more business oriented what is the outcome we want for our customers what's the impact that we want in toast business and so the types of kpis that are currently relevant to many of the projects that toast are reducing customer support tickets in certain areas of product or increasing activation and adoption of modules and here's a specific example around around adoption of modules during covert in an effort to help many restaurants survive and get through the pandemic and we gave access to multiple modules marketing online ordering and then and didn't waive the sas fees so restaurants did not have to pay a monthly fee what we find of course not surprisingly is that some restaurants thrive really make use of the software that's wonderful other restaurants signed up but they didn't actually adopt and get use and get value from the modules that they signed up for so we have right now a set of adoption metrics really trying to go back to those customers and understand and share best practices with them and share the value that they could be getting if they were to adopt the um these modules more deeply in their organization and so i think that the kpis uh that are attached to our okrs tend to be those business oriented kpis right right yeah that is i mean i think okrs are you know that's another webinar to unto themselves i think um and and uh i think you know from at least from a jellyfish perspective we will see it sometimes in in the form of you know actual development projects like the you know are you working on the things that matter most like are these are the ones that are tied to our okrs but as you said there's so many other things that can go into an okr that um you know without having like a you know i feel like i not talk much about the people oriented either okrs or kpis that we have and so in the people oriented okrs we have kpis currently a toast that relate to um hiring and managing attrition of our team but to enable that we count and measure and look for um promotion and diversity and equality or equity of of the experience that toast across different engineers and so all of those things and as well as the employment employee engagement scores that we look at and so the kpis there can be to try to raise our level of comfort or happiness or excellence in any one of those dimensions where we feel like behind yes doesn't sound like those are tracked in jira or github correct correct you know the hierarchy yeah we have some of the systems systems is what we're transitioning into for our opr's and so in a company like toast and i'm sure many of our audience members are feeling the same thing and the systems change over time just like the teams and the products and the projects are working on change over time so we're also in that case right i'm keeping my eye on the clock and i want to like field at least one more question before we run out of time and i'm kind of picking uh i'm going against the votes here a little bit um somebody asked one of my struggles is explaining to stakeholders metrics not to use how do we educate engineers that how do we educate non-engineers that buzz words like story points or bugs are not necessarily the best measure of success yeah that's such a that that would definitely be like toward the top of our frequently asked question listen i'd love to get your take on it yeah yeah i love this question um and i have been both in my less experienced mode as an engineering leader and my getting more experience mode in this place and and maybe my readiness to sort of address this head-on with my leadership team has changed so i will absolutely address it head on and say it's the outcomes for the business and for our customers that are the most important thing yeah i understand that we want to have philosophy and productivity and efficiency and excellence in everything that we do but i only want to discuss those in the context of are we having the intended outcomes for our customers in our business and so i it is very easy for teams to get sort of hyper focused on story points and i think that the fact that there are marketing funnels and sales funnels gives sort of leadership in general the expectation that you know all of these numbers are just numbers and they're funnels and it's the same i try to explain that uh you know software engineering is creative work it's team oriented not individual oriented that the metrics that we use and the way we think about this is wildly different because this is a collaborative collaborative effort and a deep sinking kind of effort and so i try to uplevel the organization i do engage for sure in productivity metrics but i try to engage with them only in the context of a bigger suite right yeah i think i think that that certainly resonates i feel like the well i don't use the wrong phrasing but like the rookie mistake might be to say like those metrics don't matter so therefore there are no metrics i think it's it's it works really well when it's like here's what you should pay attention to here's some other stuff that we are paying attention to in engineering like so you get kind of like your your show your work credit for it and you don't spend undue amounts of time either drawing attention to a thing that is actually you know very misleading for people who don't understand it and also also not the priority but like you also don't try to pretend like it doesn't exist or that it's um you know that it's not something you you don't know how to know how to keep track of yes very good and then that and that again is like a company culture thing of like you know when you know when does it really you know when do you really need to kind of like draw a hard line and say like no no we're not even gonna like talk about that versus like yeah we have it it's in the background we use it for our own operational stuff but we don't use that as the as the as the primary you know definition of success here yeah i really liked when the industry and certainly my environment changed from more of the just strictly agile sprint metrics including storyboards uh two more of the release cycle time metrics because i think that both of them are relevant but one is more relevant within the team part of the organization team by team looking at the wrong metrics and one is more relevant to the output and the outcomes of the organization and so i try to make that point that what the outside-in team needs to care about is what are we releasing how often then is it in the quality that we're looking for but yep that all makes sense um mustafa i've got my eye on the clock here it's um we're getting near the top of the hour maybe we have time for um for one more question uh there's one that that i like that is how do you adjust measuring your team to account for moving into do new domains and so i think maria you can answer that either way like because you definitely have experience moving into different product spaces but you also have to experience like kind of you know switching processes and tech sets yeah that's interesting i don't think i changed the metrics at all i think i adjust my expectations for how the metrics are going to look parallel to a new hire or the metrics for people at different levels of experience on their engineering matter and it's the same metrics it's how you contribute important how does your contribution sort of scale and trend um but it's not that we need each one to be equal or comparable to the other and so as a team moves into a new domain or an individual moves into a new domain we'll measure the same things but i think that we look in a context to well what's the trend that we're looking at first is what's the the actuals that we're looking at yeah i don't think i'm qualified to give grades on the answers but if i were that would be the a plus answer like if you're going to change things having some baseline of you know this is what we measure and care about you know setting expectations that everything's going to change because that's that's the whole thing we're either moving tech stacks moving process moving products but like having some you know baseline like this is the stuff we care about let's measure it before and after is is is super important yeah i love these questions though we can keep going all day here i know we should do another webinar this is like this is super fun um but i want to be respectful of everyone's time and uh and yeah thank everyone again like the uh you know mustafa promised me that that um people would be engaged i've certainly been on other webinars where we're you know kind of making up our own questions for each other but this has been this has been great and uh yeah i look forward to doing this again sometime yeah thank you uh very nice to interact with you and thanks mustafa and phil [Music] awesome thanks everybody