Automated selling for quality assurance

How to create outlook signature

we've been able to somewhat get to the point of being able to totally automate QA or at least have the proof of concept to automate our quality assurance processes in our contact center with all of our different clients using the actual scoring forms that our client utilizes our clients and you know this was kind of an eye-opening experience for me so I don't have all the answers on this I'm not an expert on this we are not a software company this is something that we've tried to figure out to be an added benefit for our our clients and I think the lessons that we're learning can be taken by the entire contact center industry as a whole to say hey this can be done right this can this can really be done um and and it's not as difficult I think as as as it may kind of come out to be and to be honest I think internal contact centers and bpos and and non-third parties right I think it's almost easier for us to do this is one of the few tools that I think what might be easier to do kind of In-House um or do on your own than actually trying to go out and purchase a third-party software not to say that there's not stuff out there now because of course after I post that I'm inundated with so many companies that are saying that they're doing this and they are to maybe a certain extent but I still really haven't seen the in-depth um how difficult some of our stuff could be that you really would have to figure out and I think it would be very difficult I'm going to talk more about that as we go so let me set the table here I'm live on Tick Tock I am live on LinkedIn so a couple you guys hi Sean how are you doing Kimberly already on on LinkedIn some of you guys here on Tick Tock as well I'm gonna do this as an AMA again so any questions that come up if you guys have anything that you want to talk about with this stuff I'm going to get into it how we've done it how we're doing it um and and let me know if if there's anything that I can kind of help again I'm an open book with this stuff it's not like this is going to be a product that I think that we can really go sell because to be honest I think it's too easy the the thought behind this you know it's very difficult and very easy at the same time which which I kind of will get into and again I'm kind of just just rolling off the top here I really know agenda because I think that there's going to be a lot of questions but the only agenda I want to talk to you guys about how we did this so why did we think about automating our QA process first and you know I'm a huge believer in sentiment scoring I think sentiment is as you guys know if you've been following the podcast you know can kind of trump csat King kind of trump NPS because again it's it's a hundred percent of all of the calls and it's scoring them in kind of a way that I can feel comfortable that you know we're using proper tone we're using proper word choices and it has definitely correlated to csat scores it also is correlated to um who's successful on the program from an agent standpoint so I wanted to take that to another step and and all of us have been playing around with chat GPT we've been doing different prompts for different things and you know as we were sitting here and talking through we said why can't we develop prompts that will go through our QA process do it exactly like an agent now these are in-depth prompts right so this is this is where I think the the difficulty right it's it's really easy to do if you can figure out the prompt it's very difficult to do if you can't and you don't understand the prompt or you know you're you're lacking in some of that manner which we've done a lot of experimenting with this to kind of figure that out so so the first thing that we did is we said all right we need transcripts and I think transcripts about just as a side thing transcripts you know and talking to a lot of my industry friends especially at the ccast platforms um they're going away very quickly voice is going to be coming really soon to to AI to where you're not going to need the transcript we're going to be able to do it directly at voice at you know February or I'm sorry July 31st 2023 we don't have that yet in a way that's I think affordable so what we have done is we've taken all of the chat transcripts or is We have basically all of them in our interaction analytics right so our analytics platform we are utilizing and pulling um our our transcripts now if you don't have transcripts this thing becomes a little bit more difficult right and it becomes extremely expensive as well if you have to take your voice calls and go get them translated so again I think one of the the added benefits of of having analytics right is is you do get the transcripts and we are still working through this process so let me again let me say this as well this is a work in progress this is a proof of concept this is not a finished product we're not even rolling this out to our clients yet this is just us figuring it out in in last week was like the the big eye-opener like holy geez we I think we can really really do this so again we take the transcripts and then we we started testing them on the on the chat gpt4 UI ux the desktop you know the basic you know the 20 a month thing right just to kind of test things out test our prompts out and some of these can be difficult you know if you have a QA platform that is just and we do have a couple of these with a couple clients they were the first one who tested that are just kind of yes no right did they do it yes did they not do it no right all points zero points right so a yes maybe five points a no is zero points and again we're talking about scoring off of actual scorecards that clients or you have had for the last 10 years that you like that you benchmarked that you know work well that you know if a 80 or 90 or whatever that number is you know uh 90 out of 100 you know that that's going to correlate to a pretty good customer experience and that that agent knows what they're doing so you feel really comfortable with that I think that was another problem right so many people say hey we're automating QA but they're automating it with like their own proprietary numbers they don't correlate well to you know what we have used from a scoring system so we wanted to do that so again if you have a yes no you know maybe you have 25 questions you know each of those are are worth you know you know four points right for 100 Point score um and then you can very easily just write a prompt to say hey look at the transcript answer these questions scored at the end right let me throw this too and again I just have so much on my mind this it's pretty exciting The Prompt can also give you and I mean this is obvious but the prompt tells you what you want your output to be as well right so you could just say hey go score the call but you can take that different can you please score the call can you please show the sentiment of the customer and the agent based on the transcript um can you give us you know five ways that the agent can improve um even if they had 100 score or if if you know they had a a lower score what are some of the ways that they can improve upon that right there's a ton of different things that you can ask to get outputs that would be extremely difficult to do in a time consuming way for for an agent um so taking this writing prompts easy questions that's an easy one what happens if this and this is I know a lot of you guys right um and let me just say this too the two things that you need I mean and I think that again I think all this stuff is obvious but it's not being said the two things that you need are number one is we need to be able to have access to the data right so chat GPT whatever model that you're using has to have access and be able to see it so right now we're using a transcript if you have things that are on your that they have to check on their screen right that that age is doing that's going to be hard it's gonna be very difficult so again I'm not saying this is the end-all be-all um there are a lot of things that it's still not going to be able to do but from an actual just transcript or voiced type interactions um I think it can do things pretty well so here's a more difficult use case that we figured out right we have calls that come in but there could be you know four or five different types of calls right there could be a sales call there could be a service call there could be somebody asking about a provider there could be somebody canceling right in all of those types of calls correlate to different questions that are being asked right everything from verification to um you know asking you know what actually happened on the call to disclosures at the end to making sure that everything from us from a compliance you know issue if we do have a sale for masking to doing all this stuff right that becomes extremely more robust I don't want to say difficult but I think more robust when you're thinking through the prompt right so you have to you have to have a a really analytical logical mind because again this is still a a kind of computer process but you have to have the product program knowledge and kind of the art right with that too so again we've been trying to bring in our whole team of I.T of management of the on-floor guys that deal with this every single day are obviously our QA to come up with with really robust prompts that are kind of there's a lot of if vans that are a part of it um you know looking at specific types of language if a customer says this um you're really getting in depth right into what could be happening on a call that would then correlate to a specific type of call that would then move down the channel to a a specific type of question from a QA standpoint that's the reason I think it's going to be very difficult and I'm not saying it's impossible for you know a lot of these and I did a video on this a lot of the you know kind of SAS players that are trying to do this as a business it's extremely difficult unless you want to sit down with a customer and and really sit through and talk through a prompt um I think you know there's a lot of of things that that you you really need to to look through and there's a lot of trial and error that that will go into what actually works and what doesn't I mean and to be honest you know we have seen slight changes in verbiage right do have some can have some significant changes in in what happens with the the output of the score um but I think when we got it to a point where we felt comfortable and then we would correlate it with our our QA we call them eqhp via quality shares we when we would do that um they would correlate extremely well so I think from that aspect you know we were we were doing you know a pretty good job when it came to how do we now scale this because we feel like we have a prompt that we can scale um let me stop there and and there's a couple questions here that I want to go on LinkedIn so what is the what is the what do you expect the total upfront investment to be to get this working is it already integrated with chat gbt API so yes and no I mean I'm not there yet be totally Frank we're not there yet we have connectivity um our apis are connected to chat GPT we have been over the weekend um we've tried to scale and when I say scale we try to do like you know 10 to 15 different um of the same thing we're looking at that today uh scoring 10 to 15 different calls or transcripts everything looks pretty good so far there's a couple things that we have to change from prompts from a you know when somebody calls in to cancel right we found some things that we need to tweak there um but from a cost standpoint you know what we have found is chat GPT 3 or 3.5 we can get pretty close to the same answers that we're getting in chat gpt4 the cost is crazy um it's crazy cheap when you're using chat chip D3 so I I don't have it in my note I have it in my notes I don't have my notes in front but I believe that if we were going to use chat GPT for um it I forget what it was let me let me find out there was a there was I think it was 100 to 150 um calls that we could do for like twenty dollars right which isn't a lot right when you're talking about thousands of calls um how that correlated with the the input and the output costs using the API um but with chatgpt3 or 3.5 if we can get that to work with that it was crazy it was it was thousands of calls um so that's something that you know we we have to look at but again for me you know being a BPO I think that this could be an added benefit for for our clients so if I go to my client I say Hey listen we have a way now to score score not just give sentiment not just kind of give an arbitrary number but use your scoring platform use your scoring form that you guys wanted us to do and we can score 100 of those calls right and again there's other questions that go with that like what do we do with all these costs right we have thousands of score calls but we can have that conversation here as well but I think you know from us there's a value add there that makes total sense and also I mean let's talk about the inevitability here if this does work right and I'm saying if I'm not saying that this is a hundred percent you know I I initially started this last week saying by the end of the year you know I think that QA could be totally automated by by these type of processes and I I really do believe that and you know six months of work on this I mean this thing's going to be it's for us it's going to be awesome I can't imagine if you have 200 programmers just working and focusing on this that they're going to be really good at it as well so yeah I mean I I think that there will be a human toll when it comes to scoring right I think that this is the low-hanging fruit where we could see AI really take some some of these jobs from a scoring aspect but I think I think that moves right because now we have so much more data scored so now we have to look at what do we do with all that and I think a lot of these QA guys which they've been a lot for a lot of organizations they've been doing already is is coaching right and getting better with the agents because you're not going to have to score you're going to just have to be able to look at certain things get outputs that you think can be helpful you know find certain things that can can improve the agent and then go and start really coaching and get really in depth with um with that so again from a cost standpoint I think it is negligible um when the third party guys come out when the ccast players come out I think it's going to be expensive so I mean I think it almost pays to think this through yourself um now granted it's not there yet there's security issues there's HIPAA issues I'm not again I'm not saying this is the end-all be-all but how far this is right now is pretty insane what you can do right now is crazy um in in the scoring of calls at scale if you have a good prompt is is really amazing all right another question um there's a lot that can be done in the space using chappy T is not the right thing to do as this is a generic data set I've just checked with some of our customers they have 30 plus different types of audits First Step bring all your data into a platform that can be used to train yeah and and I think that that's that's what is going happen eventually right that I think people are going to almost have their own instance of chat GPT right like you know very large companies are going to do that I don't care about the learning piece of this at this point and I think that's where we get we go too far right the the AI learning of what you want to do is something that's going to happen and it's really awesome but what I'm saying is you don't need that right now to score calls you need an AI engine to actually just score calls it doesn't have to be you know this learning about each specific agent in you know what the Nuance is and kind of getting deep into that I don't think you need that yet I mean it'd be awesome but we're just kind of taking a kind of a a tool that that we're doing that probably doesn't have to be done by humans right and we're scoring calls and then giving kind of just just some insights into what happened on that call and how to get things better so from a learning and a machine learning I don't really care to be honest I mean I don't think to to that's that's like a road map cool that's not a need I think the need is can you develop a prompt that can actually score calls like you're like a human being or as close to a human being um let me see let's see what else okay let me see on Tick Tock here if we got any any questions correct a lot of you know the same what you would think you would get on tick tock okay so I don't I don't I think what we've done is cool I think maybe just because we're only we're talking about it I am positive there's a ton of you guys out there or some of you guys out there that I've I've already kind of figured this out right if if you can program and you have the logic to program then you definitely have the brain and the logic to develop specific prompts right on on different types of audits right on different types of QA um you can score calls exactly how a human being is doing asking the same type of questions that humans are asking but you're actually just typing it out and having the the chat gbt process do it now there's security issues right now right now the good thing is it made me feel the least a little bit better that chat GPT does not model off of API connections right so that data that goes back and forth they're not modeling off of that does not mean that things are Totally Secure that does not mean we don't have HIPAA issues or PCI issues right I think that there needs to be a lot of thinking that through especially with what we're doing um to look at how do you manipulate the at least right now the transcript right um a lot of you know from a PCI standpoint we mask all the calls still you know there's customer information there's customer names those types of things from a HIPAA standpoint obviously there's there's some things that need to be pre-done on the transcript and I think I think that's an AI solution there's something there as well um but yeah guys I mean literally I mean I cannot explain that in 20 minutes that's really all we're doing is we're we're taking our chat transcripts from analytics all right we're pulling them in we're indexing them so that we know you know who was what and what program they're on and what skills right we have an API that can then come and look and take the prompt that we have created look at the transcript and then score the call and give the outputs that we want based on what the client says now here's the the next piece is and that's all we have right now right we were developing kind of a I guess a way to view right the outputs in a cleaner better way there's no apis right now that we have with nice cx1 to put them back into our qrma platform which would be awesome right and I think that that will probably happen right if we can figure out the scoring then to be able to put them back in so that we can you know maintain like all that agent Integrity when it came to to the QA um but I think that this will be an added a huge benefit for our customers I think that if you have one programmer on your staff you can figure start to figure this out to do on your own um don't get too complicated with it don't start thinking like we have to do all of this like crazy machine learning no you're just basically having a it's the same thing of of of transcribing a call at the end and putting it into a CRM right there's no learning that has to happen with that it's a job that it needs to handle and I think if you can think of of this QA piece of scoring on your calls as a job that AI can handle um you you don't get caught in the weeds then with with a lot of the things that I think people are thinking through um take that first step develop your prompt practice it on the UI ux on uh on the Chachi pt4 or the 3.5 on the on the desktop find a prompt that works right go uh go to your QA have them score the calls right make sure that you're within I don't know the one example that I gave the other day was you know we scored it as an 82 point the chat GPT or the AI scored it at an 82.5 we scored things out at 84.5 I'm okay with that right we're in that we're in that world and I'll tell you what if I have five people in our QA Department I'll score a call I guarantee you we're gonna have a lot of similarities but it's not going to be always perfect on what they what they think based on on how maybe robust that that's cornless so think that through as well but yeah so practice there find a prompt that you like Right Use the API then to to kind of put that in there right have a mechanism to to grab your transcripts which I know can be that to be honest that's the hardest piece um the hardest piece is finding and getting the transcript not the prompt right uh because it's it's not something that we normally do and if you have a transcript it's a little bit different to index it and kind of going through all that process so and then if you find a prompt you have the transcripts you can then do some things at scale and really score um and then and then use all of that data to then you know try to make your contact center better and realize it's not that difficult and while there still needs work to be done on this think about where this thing's going to be in six months just six months you know I know some of the huge players that are working on this stuff right now um and I I just I think it'd be very difficult I think you have to almost have an onboarding team to sit down and really go through you know like like appeal Sharma here on on online did you know he has a really good point you know there's you know they they have customers with 30 plus different types of audits and there's so many paths that QA can take based on on human interaction what humans can say but you have to be able to account for as much of that as you possibly can um with the prompt so but if that can be if if there's a a handholding to a prompt that everybody loves and likes and then maybe that you know they can just pass it off this this thing could be done this thing could be done at scale this thing could be done as a as a business um I know a lot of the ccast players are kind of thinking that through but from what I'm hearing there's more arbitrary just kind of numbers instead of really looking at like a BPO method of you know we have you know 25 different clients and everybody has their own plot has their own form how do we score that and kind of look through that so yeah guys that's that's kind of what I got I mean I don't know if there's any more questions on there again it's it's really cool but it's really not that difficult right there's some there's some things in it that are maybe unique to kind of get through from figuring out the prompt from figuring out how to get the transcript from what do you do with the output but that's kind of the fun part right especially that the output and kind of figuring out the prompt um get your team together have some fun with it um I think that you can you can really do some really cool things from that standpoint and I think you can get this up quickly like literally you can do this in a day in a day like you could at least be be you know practicing and and testing it on the on your prompt and you can get a prompt done in a day for something that's very basic so again it's a it's a whole new world out there you know this is maybe the first real piece of AI technology that I think you know can can Hammer some type of low the low-hanging fruit which is you know the scoring of calls right it's it's just like like resetting passwords it's like this you know just give it good data in and it's going to give you good data out doesn't need to learn doesn't need to get you know crazy into into certain things so again that's what I got if you guys have any other questions for me please let me know hit me up on LinkedIn um I'm sure we'll be posting more on this I love Chris Crosby uh he's like hey you should you know do a prompt have everything but have the outputs be like with Michael Scott from from the office right I think there's so many cool things that that you could probably do even with agents um that that could be that could be funny um that could kind of change things and and again if you think of it and I know that's kind of funny but that's how easy it is to change things very drastically right just with a couple words um just from a prompt so learn how to write learn how to write prompts learn how to get really good at that teach your kids how to do it like I'm trying to get my kids to use chat GPT all the time not just for like cheating on school right but but understanding how to write things and how to get the proper outputs you know we have to be creative and we have to change on on how we think as well so again uh hopefully that helps you guys that's that's the secret sauce again there's no there's no real secret to it I mean I think it's pretty obvious with what we were doing but I think the cool thing is it can be done it can be done with with maybe you know if you have a little bit of API of knowledge you don't need to be have a full robust team of programmers to get this done um I mean obviously you can take it to different levels but this this base level of scoring your your customers calls if you're a BPO or scoring your internal calls um it can be done extremely easily and uh you know let me know again if you guys have any questions and I'm more than happy to uh to help