Collaborate on Example of Sending Invoices to Customers for Quality Assurance with Ease Using airSlate SignNow

Example of sending invoices to customers for Quality Assurance

hey it's Joe and welcome to another episode of the test Guild automation podcast and today we'll be talking with Ben all about creating process to enable quality automation really excited about the show because I actually met Ben face to face uh clear of the highway and we would talk and I thought oh you make a great guess for the show he actually took me up on it so I'm really excited about it if you don't know Ben is the head of QA at storyblocks actually views storyblocks so that's cool it's a stock subscription content provider so there's a lot of experience in that area as well and he created the QA team five years ago after moving from a customer support team so to be curious to know how that transition went any focusing on optimizing exclusive processes to promote quality throughout the organization I think a lot of people sometimes when they think of automation they just focus in on tools and forget about the quality Ben's here to bring back the quality creating a process to enable quality assurance you don't want to miss this episode check it out this episode of the test Guild automation podcast is sponsored by the test Guild test Guild offers amazing partnership plans that cater to your brand awareness lead generation and thought leadership goals get your products and services in front of your ideal target audience our satisfied clients Rave about the results they've seen for partnering with us from boosted event attendance to impressive Roi visit our website and let's talk about how test Guild can take your brand to the next level and on over to testguild.info and let's talk [Music] hey Ben welcome to the guild [Music] thanks I'm excited to be here great to have you well I love Real World Experience especially someone that's built a team so I'd love to dive in before we do though is anything I missed in the bio that you want the guild to know more about um yeah no I think that's a good good uh General overall summary of where I am and kind of like a quick thing Quick Clip of my experience for sure okay awesome so you know I mentioned you start off in Customer Support how did you make that transition so in our organization we were a relatively small organization time kind of a startup and at that time we had a external team where we were sending out for we were having QA projects that needed to be done and as part of that our engineering team was going over to them and saying here are the requirements we think need to be met please check it out but we kept finding that all of these bugs kept coming out and I was I was leading the customer support team and so I would get customers who would reach out and say hey this is broken or this isn't built properly or I'm seeing an issue on your site reflectively I was a person who kept reaching out to the engineers and saying something's not happening here we're missing this the quality of the product we're seeing the customers when you all deploy these features isn't what we expect what's going on and so as we are expanding out of our engineering team the organization said hey we have this opportunity we want to bring our quality internally versus having as an external company we hire would you like to join and at the time I had no experience on anything related to software testing QA but I was like this seems like a really cool idea I'm good at finding these bugs maybe I can help them find the issues and so it became an open role I started out as a team of one and literally they just sat me down and said okay go and so I started by looking at a bunch of different resources some of them starting out were things like Tesco things like the ministry of testing where I was looking for resources to say like what should they do how should I do it and what processes should I have and kind of develop from there so I guess where did you start then I mean you had to find out you just jumped into it and also you had all these options it was there one area you said okay let me at least start here as the building block and build off that yes so at the time as I said we were receiving a lot of issues and bugs and what I found out is we didn't have a process in place for reporting bugs for managing bugs or a way to easily follow up on them so I started out by first identifying what and defining what types of bugs we have what are the common bugs we have what are the issues and I assigned priorities and slas to each of those so I met with all of the individual team leads and I sat down and said okay look these are the types of bugs we see coming in here are our internal and external stakeholders and here's how I believe based on what I'm seeing and what I've heard from customers we should map out the critical the highs the mediums and so on and then I started tracking those and saying okay are we seeing a reduction in bugs by introducing some form of tracking and issues I also began to do the manual QA and and gives some insight into the requirements so previously a lot of the requirements were a fairly blank ticket do this action and when I pulled that up it was it was basically like well what does that mean what do you want us to do here and so my push for a lot of the team was we need more detail we need acceptance criteria we need user stories we need to have some form of information that when an engineer goes to create this feature they're able to then look at that and say okay I've done the definition of done I've met the criteria and then when I get it as a tester I'm able to say okay you've added the correct automation here the manual testing looks good everything looks okay now we can ship it to the customer love it so what I really like is once again you start off with customer support and that's what I think a lot of people miss out on they forget about the customer and then it actually that's where it all starts I love how you actually were influenced by what actually happened to your customer in production and you can actually do like a one-to-one map and probably like Hey we're not just a cost center uh We've increased customer satisfaction by 10 or something how do you maintain that though so it's been five years since you've been in to the customer support how do you still get that customer feedback then to re-educate what's going on with your your quality process yeah great question so since that time we've added a research team so I have a really direct relationship with the research team so a lot of times when we have features we're making sure that the features we're implementing are things that the customers want to need and from a quality perspective we're making sure that we're discussing like in in oftentimes in research we find that customers will bring up once desires and needs which translate to me often to some form of an issue or something they experience on our site that hints at a need for something that they want so they might be going and looking for a piece of stock content and in the searching they say I don't see this functionality here and to me quality is beyond the scope of just the testing it's also the product itself and are we meeting the needs and the whys of the customer when they're coming to our product and so keeping with my research team has been super helpful they keep me in the loop the things I hear I also every other week I spend a few hours sitting now with custom customer support team and talking to customers that I think has been the number one thing to keep me in the loop so I think in often testers especially if they're siled in their team sometimes they neglect to to talk to the people who they're delivering the product for and when I sit down with the customers I can get an idea of what are the things that are actually bugging customers because I found when we started reporting on bugs we'd often have these bugs that were reported internally some engineer or product person or someone goes to use the product they pull something up and they're like oh this is broken but customers never reported that issue customers didn't care about the specific thing but the thing when the the page that they're using most frequently the tool they're using like for assets things like adding items to folders or they're doing a sign up flow when that's broken when that's not clear when the design isn't laid out properly that's when I think it has a huge impact on the customers and it helps kind of educate well what should we have testing for what should we be looking for what is the high risk stuff that actually brings the business value and thus allows us to have high quality well love that approach I just work for a Healthcare company we used to make Radiology software and they used to have like almost field trips to hospitals and actually watch Radiologists using our application and you gain so much from doing that so that's really a great great uh tip I think a lot of people miss out on how do you get that buy-in though it seems like you almost have a culture then that accepts that this is the way it's not just a check mark on the definition of done is hey is this actually going to help the customer and then you're able to measure did it actually help the customer how did you how do you become just a one one person show then then kind of pushes that throughout the organization I I think you have to be okay to allow people to ask the why why are we doing this what is the purpose of this and within our organization we're fortunate in that if I go to my product person and I say like what's up with this does this really meet the needs of the customer like is this something we want to do they're willing to accept that so I think it is a cultural aspect it did take time to build you know starting out when I started I had no experience and sometimes I'd be like I don't think this makes sense and there was pushback it'd be like okay what's going on but I I was able to use data and kind of prove the case out using now I use the research team to say okay well I don't think this necessarily is something we should be doing I don't think this is best practice here's some information showing that's the case it's more data driven in the sense and that allows us to look at that and say does this make sense and as such it kind of builds over time when one team starts to do it we show that it was successful that it had some positive impact on the organization and then other teams start to say oh what's going on there I see you have less bugs I see there's a higher quality that you have more automation that something is happening on the practice side we explain what it is and it kind of takes like an A B test philosophy that expands to have something across the entire organization nice so besides listening to customers are there any tools or techniques you use in production like do you look at Google analytics or heat Maps or anything like that to help you get an Insight before people actually complain yeah so um there's it's been a journey starting out we didn't have anything over time we've kind of developed tools and there have been ups and downs there have been times when we've had too many tools so there was a time where we had about four observability tools all of it did something slightly different but none of them did the thing we actually wanted to do and there have been times where we don't have enough tools and so issues will occur and we're like what's happening so for us I really try to lean on the data and the observability look at things like events so we can look at events we can have anomaly detection in place we can say is something happening that isn't expected we also try to see if something happens on the site like if we get an alert and we have an incident that happens sometimes the cause is for us we have a product on the site but we also have an API product we've had issues for example where a customer goes to hit or say really hard they download a bunch of content they search for a bunch of content something like that and it causes latency on the actual product that are not API customers are using as part of that I think you have to take a step back and you have to retroactively say like if I wasn't aware of this if I'm hearing about this first from the customer seeing it cause an issue on the website why was I not aware and as part of that we try to retroactively add some form of monitoring observability in place to say this is a specific case that we saw but we may be seeing it elsewhere why don't we add something in place to make sure that we get a report an alert when we see the spike it also allows us to have conversations with those customers about like well what are they doing should they be doing this what's happening and so we can kind of monitor the behavior and go from there so once you find the issue then based on the customer's feedback how do you feed it back into your development system is it open up as a bug or do you start a new Sprints I hear some common issues we probably want to get better at what does that look like so right now my team is doing a lot of Triads for these bugs that come in we were finding before that when we didn't have previously it was me I was getting bugs they were coming in I was sending them right to the the PMs and the teams but oftentimes the details weren't there it wasn't sufficient and so we found we had to have a layer in between and so when the bugs come in my team will do some degree of triage and we determine is this actually a problem that is going to be worth addressing so it's a cost to fix the issue worth the effort we have to put into it so let's suppose from a business perspective you know it's expensive to fix bugs so if you're throwing engineering effort you're throwing design at it stuff like that you have to factor that in and then from the perspective of risk is this something that is happening to a single customer is it something that's going to happen to a bunch more customers and so we triage those bugs and then we put them into the the team's backlog and then the product manager for that given team takes them and puts them into the next spring we generally are trying to address as many bugs as we can in a given Sprint that has been a bit of a struggle so for example when I started I was reporting bugs I was just putting them there is a backlog of hundreds of bugs that we'll never get to so now we try to have more of a mentality of if this is truly an issue we triage and we determine is worth investigating and putting the effort towards it's much better to do that while it's fresh while you have the customer data while you have the information rather than sit on it and then either never get to it or have it something that you know when you go to address you lack appropriate information for I love it so I'm just curious to know I've circled a lot of people sometimes when people start like a quality process or a quality organization all some people are oh they're that team's responsible for Quality they're responsible for testing we could just focusing on development who does the testing in your organization now once you identify the bug you put it on the Sprint is there a tester in the Sprint team or developers know okay that I have to test this feature before I release it or put it into uh the code so right now we have my team is myself and two other testers we have those testers embedded on the major product teams so we have three major product teams I'm on one and then the other testers on the other two when we started out we found that often I I and thus my team was the group doing a lot of the testing in that we were helping kind of write the test we were doing the manual testing we were making sure everything's in place and checking all the boxes so we're kind of a bottleneck in that regard over time what I've done is as I've gotten requests I've said to the engineers I'd like you to show me what you tested and what's interesting is it started as a conversation of like what what do you mean what okay I did this and this okay but did you do this as well I then pair with them and I walk through this is what I do this is how how I go about making sure that we check the quality boxes what's been really interesting is over time I found that when they approached me and they're like hey I need something to be tested or I need some testing to be done with this manual automated otherwise they've done a lot more they've tested the majority of things and so often my testing is more just exploratory I can go ahead and explore the product and the edge cases that they don't necessarily think of because of my product knowledge and then we've also done a lot more group QA so I lean heavily into Ensemble Ensemble testing and pairing and so a lot of the times when we have a feature we need to test we have a group session so if it's a bug that is really impactful we'll throw together a quick meeting we'll get a PM we'll get an engineer we'll get any additional stakeholders where it may be relevant we'll just make sure we check the boxes we've made sure that everything is good before it goes out so it's kind of a mixture of others in the organization testing with me kind of being the one to help move that process along so that in the end I can do more of the types of testing that we don't have in place now nice a lot of times people say oh developers can't test but I love the idea of peer programming period testing over time it's you're educating the team to you know fish for themselves so I love that that approach any uh tips for peer programming or peer peer testing uh do you have a script or uh is there something you've seen work better than others yeah I think the the two big things I would say is one ask what their thinking is when they're writing the code they're writing and how they're developing it sometimes I think if you don't express the why an engineer could be developing something in a way that meets the requirements but doesn't necessarily make sense like maybe it's a really roundabout way of doing it or maybe it's something where you look at it and you're like okay this doesn't function like I think it should so asking why kind of gets into the thought of like this is why I'm developing in this way and sometimes you find things that help move that along and help you kind of learn and understand which you can then approach and say okay I would do this differently this is how I would I was thinking this feature should act and so what's cool about that is I think sometimes it highlights there's a gap that maybe you don't have all the requirements you need maybe there's something that you're not seeing or you need to have in addition to what you currently have for that ticket or feature and therefore you can highlight is like hey if we want to do this we should be adding x y and z and then not being being afraid to say I don't know so I think there are a lot of cases where something comes up I I have examples myself I'll be working with an engineer on a feature they'll ask how something is supposed to work or they'll ask like from an automated perspective how I would handle it and I'm not sure especially with new tools that you might have or new Engineers who come in from a different organization they might have experience where they do things differently granted we try to have a general process that is similar across all teams but you know they might bring their own blend to it and if I'm not sure instead of I think trying to work it out you know figure it out and I guess pretend that I know what I'm doing I like to just stop there and say I'm not sure let's figure it out together and so pairing on that and figuring out is the solution what we think it is or like how should they behave let's go check it out and ask the EPM or let's go look at the product ourselves and say what's the current behavior what should this be when it's implemented or let's talk to customers let's you know talk to the research team and say hey we don't know how this should work do you have any input like was there something that a customer said before that might give us some feedback on this that's awesome advice so you did mention automation I know automation isn't necessarily your area of expertise but how do you work to support maybe the teams in this area I think the biggest thing I found is having a some form of a strategy in place so I think having good documentation that gives insights to what types of tests your organization uses kind of like good examples so what should a good unit test look like when should they be applied what is what is the difference between using a unit test and an end-to-end test for example I found that often Engineers just aren't sure and we have all of our Engineers writing our their own automated tests but we were finding a gap when I started where it was like I'm writing a test but I'm not sure which level I should do or what type of tests I'm going to take my stab at it and sometimes that can result in a test that doesn't actually do what you intended to it's not work frankly it's not testing anything at all um you know we've had tests where it's just like look at this page and it's like well that could be done at a different level I think pairing with and reviewing the requirements that we have has been really beneficial so if you have some degree of a understanding of the acceptance criteria for this ticket and you've reviewed it as a group so if you have a session where you're looking at the ticket and you're figuring out what's supposed to be included if you review it as a group you have a clear requirement of what's expected as part of that we have a conversation around what is the critical part of this product where if this broke and I went over to the PM and I said hey this is broken they would say we should fix that right away so having that conversation is really beneficial because it allows us to sit down and say Here's what needs to be tested here's what needs to be done from an automation perspective and then having Tools in place that Engineers find are easy to use so for example when I started we are using selenium for all of our end-to-end testing selenium is a great product but because of the work it took to create the test and kind of like the knowledge it takes I think to use it to some degree a lot of our developers were taking very very long time to write tests and so there'd be cases where they just didn't write a test because it was too hard and so we about a year or two or so ago we moved to Cyprus because we found that the developers were able to write the test a lot more efficiently had a better understanding of it and while it's not exactly necessarily the tool I would love to use myself I found it was a tool that was efficient at getting the job done for the audience that was using it for the developers and so I think thinking about what you're using and the audience who's using it can sometimes help you think like okay I might have a preference for this product but if I'm not necessarily the one in the weeds day to day getting in there and you know writing all the automation it might be better to to find a product that works for the group that's going to be using it a lot that's Guild gold right there that one tip for sure would save a lot of people a lot of headaches I love that how do you keep on on board then or you know I met you at a conference so how do you keep up the latest trends and developments and quality and Process Management are there any other things specific that help you stay up to date on what's Happening yeah I I try to I try to follow a lot of people on LinkedIn and Twitter and the like so I'd like to see what what are they talking about what is the thing that they're putting talks out about but also I look at slack forums and things of that nature to see like what are people's feedback they give when an article comes up and they give comments sometimes I think there are a lot of people in conferences but sometimes those are the people who are just willing to they're extroverted they're willing to go out they're willing to talk not everyone's like that but there are a lot of introverts I've met in in these situations where yeah you know it's like you have great ideas and so I think just kind of opening the channels and and seeing the people who are giving feedback on the things they might not always be the people who are the loudest voices of the people who are at those events presenting but they often have great ideas so I try to talk to fellow testers um for me Ministry of testing was quite possibly the biggest jump when I started testing as I said I had no idea what to do I literally Googled how do I do software testing um I knew how to find bugs I knew how to do manual testing but that was about it and I found the community there really expanded my knowledge and so it set me up with podcasts like test Guild where it was like oh cool okay there's this really interesting thing I want to listen to and I try to take the ideas that I find from that and I try to share it out with my community in my organization and get their feedback uh I also am a big fan of well it may not necessarily be like the latest Trend I find a lot of people who come into my organization work at organizations a different size and scope and have a variety of tools they've used so one thing we like to do is we like to sit down with our new hires you're an engineer I just started we have a quick half hour meeting we introduce our ideas our principles what we do on a quality like how we go about handling quality assurance for our organization well we also asked a question of tell me what you did how did you make sure that we checked all the quality boxes of your organization walk me through your feature development flow and what's cool about that is sometimes I'll find a tool I had no idea existed I'll find a technique that I'm like oh okay cool like like risk for example wrist storming was something that I didn't know about right away someone mentioned it when I first met them when they joined the organization it was like oh this is really neat Ensemble testing same thing someone brought it up as a technique that they use as their org in order to expand their testing Scope when I investigated it was like this is really cool let's try it out onboarding I love that a lot of people probably miss out on that I haven't heard of anyone that really does that so that's a great great piece of advice as well so how do you come up with this you just been experimenting it just uh you just try things and then just iterate because a lot of times I see people like this is the process and this is what I do and they never deviate from it you know yeah I think a lot of times in organizations we like the a b testings for the customers but we don't like the a b testings for ourselves so I'm a big fan of just like trying something out what I like to do is I like to approach a team who I think might be open to the idea I like to take something that I think is an interesting tool that doesn't have too much of a lift or a level of effort and I like to say I'm not going to ask you to commit to this what I'd like to do is see if there's a problem we have it could be anything from issues with automation to uh how quickly we're able velocity for deployment and I'd like to try this tool out for a few Sprints and I want to see if it improves and if you have positive feedback if you have positive feedback maybe this is something that we should use across the organization and so I think you just have to like don't get too married to one tool don't get stuck on one process be willing to try something out I think it's also awareness of the problems that you have in your organization so if you're seeing that issues are occurring take a step back and figure out like take a step back and figure out like what's happening I the way I put it is I tried to take a perspective of if I came into this organization right now and someone said you're the director of quality what would I do how would I do things differently what tools would I introduce what processes would I try out and if I have those what's the blocker to me trying them out in a lot of cases teams are open to them because they feel the problem is that that tool or or that new process would solve and so we tried them out as an A B test we see if it works and it has improvements and if it does great we find ways to implement that tool as much as we can cool okay Ben before we go is there one piece of actual advice you can give to someone to help them with their quality testing efforts and what's the best way to find or contact you yeah so I think the biggest thing that I would say is to over communicate and communicate often I think a lot of times testers get siled and they become the voice that doesn't really speak up very often but they also then sit and say oh man I'm feeling all the pain from this thing that's happening this process or this tool or this thing that this engineering did I think if you speak up and you communicate about what's happening and this could be anything from ideas I have to problems I'm feeling on my team to issues that I'm seeing on the site sometimes others also feel those issues and sometimes you might find a common bond with someone you didn't realize you had and so I I also like to say if you are over communicating especially if you're pointing out a problem try to come in mind with a solution so I don't know if you've ever been in those situations where someone comes to you and it's like this thing is broken okay what would you like me to do with it I don't know I'm just letting you know it's broken but if someone comes to you and says this thing is broken I would like to try X it has a lot more of an impact and so I think if you're communicating about problems you have think about what Solutions you might be able to use and that doesn't necessarily have to be something that is you know fully fleshed out it could just be an idea I want to try this tool I want to change the process a little bit minor changes go a large way and then uh to contact me LinkedIn I'm on LinkedIn I'm on Twitter or X would be the best way but I think LinkedIn is a primary there thanks again for your automation awesomeness the links haven't done a valley we covered in this episode hand on over to testguild.com forward slash a460 and if the show has helped you in any way why not rate it and review it in iTunes reviews really help in the rankings of the show and I read each and every one of them so that's it for this episode of the test Guild automation podcast I'm Joe my mission is to help you succeed with trading end to end full stack automation awesomeness as always test everything and keep the good cheers thanks for listening to the test Guild automation podcast head on over to testguild.com for full show notes amazing blog articles and online testing conferences don't forget to subscribe to the guild to continue your testing Journey