How can i industry sign banking new york presentation free
good morning this is Mark trash with interactive services and welcome to this 40 minute presentation with Matt PLAs the chief Learning Officer interactive services and then we have two guest panelists Kim and Dawn who heads up compliance learning for city worldwide and her Pierre Gary Collins who is responsible for leading compliance learning globally at BMP by bus it should be a great discussion in terms of what we'll be walking through today for the past 18 months we've brought together some of the world's biggest banks to meet face-to-face for two hours three times a year here in New York to talk about compliance learning and it's been an opportunity to share what's happening war stories insights plan for the year ahead but one area that came up from the participating banks that participate in the roundtable here in New York was the need to benchmark their program with peers there is no clear insight in terms of how long an AML program should be what languages it should be offered in so with that in mind we ran a survey to our participating firms who are all made of large international banks where we asked them a series of questions the data was collected anonymously so it wouldn't be tied to any particular institution but off the back of that we got some great insights on potentially what represents best-in-class burning a compliance program at a large international financial institution we hope this discussion today will provide you good insights if you hold a compliance learning leadership role at a large global bank or you're joining from a small regional institution I want to hear how some the bigger peers to do it or if you're involved in another industry and want to see what's going on in financial services just a final piece of background around the firms that did participate they represent some the largest global financial institutions collectively they employ over 500,000 workers worldwide and their businesses spanned from banking asset management broker-dealer and advisory services with that I will hand it over to Matt Kim and Gary for what I hope will be a useful discussion we will be taking questions throughout so if you wanted to ping those in Mike colleague Neil Cullen who is overseeing the presentation will funnel them back to our panelists so that we can enhance them Gary and Kim do you want to just quickly introduce yourselves and just give us a very high-level overview of the specific roles that you play at BMP pieman city respectively you can hear what I go oh okay sure I'm sorry go ahead Kim and then I'll I'll go Matt okay great well welcome everyone in hello I'm Kim Van Dorn I am the global head of compliance training and strategy at Citigroup been here for a little bit over a year but have had you know quite a long tenure in the compliance training space so I'm looking forward to this discussion and talking to some of the points of the survey and doing that along with Gary who I have worked with in the past so thank you all for coming and thank you for having me great and so I am Gary Collins as you just heard I lead both the training program and policy governance for the Americas region at BNP paribas I'm somewhat new to the firm I just coming up on my one-year anniversary and one of the charges that I had upon coming to being deep area bob was to remake reimagine a learning program so Interactive's kind of been a great partner with us and helping me along in that journey prior to BNP paribas I was at GE Capital where I could have the pleasure of working with cam and before that I was a federal prosecutor at the US Attorney's Office in in Washington okay thank you Gary and thank you Kim and thank you mark we're going to get into the content of the report now and I'm sure might mention this earlier one who doesn't have a copy of this report will be sending it out afterwards and you're very free to ask questions throughout and we'll try and answer as many as we can at the end I'm going to walk through each of the insights we have seven in total that we pulled from the report and I'll give both Gary and Kim an opportunity to chime in if they have any thoughts on that and if not you know feel free guys just to say no comment and we'll move swiftly on so of the seven insights some of them you may find that the fighting findings confirm what you already know others may come as something of a surprise and I'm sure Gary Kim are going to be able to give us some color commentary the takes us behind the scenes obviously it was a fairly small sample but a tomato point while there were only ten phones involved it actually represents over half a million workers across 71 countries so it's pretty representative of what's going on out there in the terms of in terms of compliance training so let's jump into the first insight it around language what we found is most organizations global organizations find that English is increasingly dominant as the business language and that another average of four possibly five languages including English would account for up to eighty percent of their population one of the conversations we often have with our clients is around how many languages will we translate our content into and who should pay and one of these the findings that we took from this survey is that perhaps the the corporate head office should be looking at providing the content for that 80% the English cluster for languages and there is an argument that says that other local markets might want to provide their own versions of that which could be controversial but it's certainly something that some of our clients are starting to look at now taking a little less responsibility for that entire global footprint that was the first things like we drew I'll just say if either Gary or Kim have any commentary on that Gary how about yourself yeah so I was just kind of show up by saying this is undoubtedly the direction that we're moving in we're all in environment where we're trying to do more with less and you know from my perspective we're trying to simplify everything and that includes being more thoughtful about really kind of what the need is and in terms of the number of translations so what we've done here is we've kind of first you know assess where are the regions in which a particular language is mandated by law so I think one that's most common in the Americas is the French first rule in Canada where a lot of that has to be translated into French but we certainly have become we've narrowed the scope in terms of like what is absolutely positively needed and started the process I can't say were there yet Kim and city are probable but ahead of me in this but started be the process of a Center of Excellence which sits in the headquarter function which would execute those on behalf of the regions and the territories okay thank you and Kim have Excel yeah excuse me I agree with all the things that Gary just said but specifically you know the there's more attention being paid to local requirements with respect to training and translation so we're really trying to be you know very focused on that and really understand the landscape country-by-country here and what's required locally but you know it really gets to the importance of the effectiveness of training and how do we measure the training is effective in certain areas in certain jurisdictions or markets it's important that we translate so that we understand that the training or know that the training is effective for the learners that are taking the training so I think that's another important aspect of it yeah and interestingly what this study didn't really look at was regionalization so actually adapting the content to fit the region not just the language so that's a whole other area but certainly I think Kim you're hinting at that that kind of risk-based approach in terms of who needs what when rather than you know and perhaps a move away from the blanket approach we've had in the past we're where L want force and you translate it for absolutely everybody great great okay we're going to move on to the second inside so this is really around the volume and the question we asked is are you giving your people too much compliance training or not enough compliance training and we had some interesting results there was a median of around four hours which felt felt about right there were some out lives a couple of organizations down to and one organization providing ten hours of training what was interesting to us was that the the ten hours the longer pieces didn't necessarily reflect the complexity of the organization so the more complex organizations weren't necessarily giving their populations more compliance training I think what it tells us considering that the regulators for these organizations are almost having the same is that to some extent it's arbitrary and it's a decision made internally and it's something that we do perhaps have a little more control over than maybe we thought before so certainly very interesting to see the four hours which we expected for the median but also two hours which seems which seems quite short and then ten hours which seems very very long for the the training so I'll throw that over to Darren Kim for your thoughts on this piece just around how much is enough how much is too much maybe Gary again we could start again yeah bulletin so for our firm were clearly on the higher side of that were clearly kind of much closer to the ten you know I think the challenge for us is not necessarily the number of hours but kind of coordinating that number of hours with the two or three different levels of training that happen inside my particular organization so if you think about the Mt carry bar headquarter function is in Paris you know at the regional level of the Americas level there's another headquarters and then all of our operating entities they all have training programs so our challenge around this is we're definitely on the high side but in terms of managing and coordinating what's coming a buck from to us from above and what's happening below making sure that number one it's not duplicative and number two it lands in a place that we think is about right and I think for us right now that 10 hours is probably about right you know one of the things you know it's interesting and you know one of the the benefits of having these discussions are just sensing and understanding what other folks were doing so with that I'll turn it over to you Kim I don't know if you get additional thoughts there yeah a few other thoughts in addition to the just the coordination of the regional functional and global pipe training to make sure that there's no duplication that's a big big piece of it and that can be solved by really establishing a robust training strategy globally which takes into consideration a very well-defined well map well plan training development process training develop training plan development process that is if you have a good process and you spend the time upfront then you can come up with a strategy that helps kind of eliminate some of the duplication and gets everybody on the same page and then the other piece of that is how do we train what are we considering training is it all training just through a learning management system are we taking a different approach to or as you mentioned earlier more of a campaign approach to training and learning and are we making it a more holistic thing than all just using the learning management system there are other ways to train and what is the strategy for a topic really look like and that can kind of help pull back on the number of hours that people are spending you know sitting in front of their computer so I think there's still a lot of work across the industry to do here to bring these numbers down because you know most of us are kind of on the high side yeah Thank You Kimber and actually takes us on to the next the next piece that we're going to look at which is about the duration of courses and I think you know you mentioned the campaign as we start think in a slightly more strategic way about how we communicate training and awareness perhaps the core training piece is allowed to then get a little bit shorter and which I think would benefit everybody and in fact the third insight that we had is really around this duration of training so there is a pressure across the training industry not just compliance to reduce the length of modules and I think learners particularly millennial learners are telling us that they like short bytes of learning they like five minutes they're like 10 minutes they don't like 45 minutes they certainly don't like 60 minute and yes there is Memphis mandatory content you have to allow for a certain amount of content to be delivered but the individual programs we do think are getting shorter and there's pressure from the business as well you know how much time do you want my people wait in their desk taking this training so those those that pressure coming from to end when we looked at the average training time we found it was around 33 minutes and then if you just scroll down a little bit think we've got a table a graph there that illustrates them the durations we found so having chatted with some of the people involved in the survey I think the general consensus is this looks about right so the retail product pieces is an anomaly but something like an D month anti-money laundering cuff services of 42 minutes being the longest of the mandatory modules feels about right and with code of ethics close behind data privacy hard on the back of that and then maybe down at the other end somewhere between 10 to 20 minutes on social media again it just instinctively feels about right I think there are some questions we can ask about this I think if we if we see that we are spending 36 minutes on our entire anti-bribery training and we're spending almost that much of 26 29 minutes just on our gifts and entertainment training you know maybe that's maybe that's appropriate or maybe one of those modules is too long or the other one it's too short so it's interesting to see how these panned out in terms of the duration of individual modules again maybe I've asked in first on this one what do you what are you seeing what are you hoping will happen in the future well I'm hoping a couple of things I'm hoping that we'll continue to thrive towards having smaller you know modules of training and bits of learning that people can kind of do at their own pace I think the one thing that will really help us in this space all of us is to embrace mobile technology even more because what I'm hearing from most people is I'd like to be able to do this on my own time not have to be tied to my desk use my iPad or my phone and get through this and I think so that's that's the accessibility to the training is one thing and it's easier to deliver small bites if you have training that's more accessible but definitely breaking it down and spreading it out is definitely something that is desired by most people yeah absolutely and in fact on the last webinar we gave we was talking about mobile and we talked about the fact that a couple of years ago mobile was about different audiences so sales guys on the move for example and now it's all of us at different times of the day so we all check emails on on our phones first thing in the morning in the canteen end of the day it's been a position to take small bites of training on a mobile device but it as soon as you start going above the 5 maybe 10 minute mark maximum it starts feeling very creaky on that smaller device Gary did you have anything to add into this piece yeah this is one I feel real strongly about this just to take a step back I spent a lot of time at this cam like a reading about how people learn and how they retain information and I kind of walk away from all that academic scholarship and in a place where I firmly believe that shorter is better shorter well done is better Abraham Lincoln was famous for saying um I would have written you a short a letter if I had more time I recently went through that exercise with Interactive where we had to do a confidential supervisor information module which is not something you can get off the shelf and when we built it on the
goal with it for the goal was for it to be eight to twelve minutes and it's a very complicated topic so in terms of the design we probably spent you know thirty percent more time on the design of building something shorter that is comprehensive so it actually takes more time to be thoughtful in doing something as comprehensive and that's shorter but it can be done you know interestingly I'm working on an anti bribery corruption module right now and it's landing I'm hoping it's landing between 25 and 30 minutes which looks kind of close to what we see here yeah fantastic and I think it's partly it's partly about mobile it's partly about giving people access it's partly about the culture as well and there's one of the questions we'll look at later it's about tracking - it's about to what extent are you happy for people to take this content without you necessarily having a 100% view of how long they spent in it how well they did did everyone take it at the moment our technology is perhaps not quite at the point where we can deliver this array of different formats and track each one in quite the way we'd like to right now maybe we could move on the next one's interesting it's about testing out testing down so for anyone who isn't familiar with this concept the idea is that if I if I am able to demonstrate an existing knowledge or ability by taking an exam and passing it then perhaps I get to skip or clip out of the training interestingly 30% of organizations we spoke to already do this in their compliance training a 40-plus percent plan - and only 30 percent have no plans to and I would say - that 30 percent don't plan to inmate you may well find it's coming anyway because it's increasingly populating the training world and you can imagine why it reduces the opportunity cost loss and from people who are taking training when they don't need to be it pushes people through their training in a lot a lot faster and it allows people just focus on the pieces they don't need so a perfect piece of test out software would test your existing knowledge and then only give the content that was relevant to the questions you don't fit in correctly that's now very easy to do it wasn't so easy to do a little while ago so it's becoming more and more common I suppose if we take if we take one keynote from this if you don't already do this you probably assume will be some organizations are very culturally ready others less so and I would say the same for regulators from from what we're hearing anecdotally this seems to be an argument that can be won sometimes and lost other times I'd be interested to see how both Kim and Gary approach this and the extent to which they think BMP paraben City are ready and happy to have a compliance training reduced in with this particular method maybe we'll start with Kim on this one so I am a big fan of testing out and testing down particularly for topics that are you know a better fit our annual topic so your AML financial crimes topic a lot of companies do that every year that's the same same same course trait some of your code of conduct size courses I think it's a great opportunity to give people the chance to not have to take the full course but to be presented with not only questions that they should be able to answer but also some highlights so what happened in the past year since you took this force last time and what do you need to know and the great thing about it is that in addition to giving people that option you know from an effectiveness standpoint you can also measure how people are doing with the test out how many people actually were able to test out I'm an example has you go on and take the full-blown course you know we're constantly being asked to demonstrate effectiveness of learning and I think this is just one of those situations where you can extrapolate some metrics that can help help you not only plan your course for next year but also to say you know 70% of the people were able to test out some knowledge is there and the other you know 30% needed to take the test or by first up 30% of the people were able to test down and the they're 70 had to go through the course so that tells you something about the subject matter and the audience so I think like I said I'm a big fan of using yeah and I think not only does it show you where your where your audiences potentially it also shows you how good your training is because if exactly if everybody's failing a particular question or maybe that's a learning need or maybe you wrote the question that that could be written a lot better yeah and then that'll be built into your strategy for the coming four coming here absolutely Gary any thoughts on this pace yeah so this is a place where you know I'd like to get us there we're not quite there yet I'm kind of my I think my challenge is clearly my constituents so the people were actually taking the training something they're very enthusiastic about and want to see happen there's just no question I'm in a place now partly because of our firm has paid a significant sign in the last 24 months that our regulators are still looking closely at how we train on certain subjects and and I think I oppose this probably too strong a word but I'm not sure there is enthusiastic about the test out option as I am at this point so for us it's going to be a little bit longer process yeah but you feel it you feel is inevitable yeah I mean I think you know again probably at this point as mentioned effectiveness of training two or three times you know that's really ultimately where this is all gonna is being driven to like measuring effectiveness is probably the single most challenging aspect of any training program the metrics are really like we did it and when they completed it whether it's effective is hard so to the extent that this is providing a window into that there's no question it's valuable and again you know from a time perspective I think it's helpful so I think everything is moving in this direction I think some of us are going to get there faster than others yeah yeah absolutely thanks guy so let's move on to the fifth piece fifth insight that we drew out of the research and it's really around the role for ilt so for classroom training now interestingly when we when we have a look at this graph down the bottom the split between e-learning and classroom that's that's not about smaller populations who focused specifically on compliance receiving additional classroom training this is actually the number of organizations or potentials that rely purely on e-learning versus classroom two so 18% of our respondents are doing all of their compliance training they have a version of it for the classroom not just to be a learning now obviously elearning has always given us the ability to track and score in a way that the classroom struggles to do so it's easy to see why core compliance programs are put out online but again it comes back to the campaign it comes back to the risk-based approach it comes back to delivering the right piece to the right audience at the right time at this role for the classroom I'd be interested to see maybe Gary will start with you just just where you see the roles of the classroom in delivering compliance training yeah we're you know I'm personally a big believer in this it's harder because on the number of different fronts I just believe the live training that's well done is probably a little bit more effective and then eLearning well done typically means a well done scenario based learning session it's you know the beginning of August and we probably have trained over 3500 people live and we'll probably end the year just north of 5,000 it's more expensive again more difficult to do and then thirdly it's even more challenging to measure its effectiveness but anecdotally kind of what I see and kind of what you know my impressions are if you can do something live that's well done it's probably a little bit more effective it's much more expensive the execution risk is just a lot higher and it's just all-around it's harder to do but I'm a big believer yeah I'm tech Kim I think you have something some yeah just a couple of things one I just like to say that for everybody here this Gary is Gary put together a brilliant in-person course that we did while we were at GE and it was for senior leaders and the impact of us doing it that way I mean it was just it took a long time to do but the impact was just phenomenal in the level of engagement was was just brilliant so I never forgot that Gary but it is you know tactical role for instructor with training I think it gets back to your planning and you know coming up with a plan that really identifies training needs to the point where you can say okay these are topics that may be role specific role based type training needs that we have so that we can identify you know the situations where we've got to do something smaller something more engaging just to make sure that depending upon your role your responsibilities that you know the training is effective and look the bottom line is you need to walk away from training and say what what does this mean to me what does this mean as far as my job is concerned so we need to look for those those instances where the instructor for training is really effective and really really important yeah I just think that just another quick sauce to augment what Kim said I'm starting to believe that lies training I agree with what you said about roles but why training to the extent that you're investing in it I think it's you know I think the folks should probably be on your senior leaders because you know with that with that population you're trying to do something very advanced and that's very targeted because most of these people Barret have a working knowledge of the compliance framework and how it works together and I think the things that you want to be covering with that group are like or creatively how do you drive the right you know conduct culture and behave in an organization so I think those types of issues are more conducive to that life training and maybe you know if you have a limited budget that's where you focus it because again it's Kim said it's a really hard thing to do yeah I see that that leads us very nicely into the next piece which I think you've kind of answered but just before we move there just to let you know we're seeing questions coming in and we are capturing them and we'll try and run through as many of them as we can at the end we just got two more insights to cover and we'll we'll hopefully leave a good five minutes for questions at the end so so the the sixth piece we asked about was aside from the formal training which in the current current world really needs to be done online so we can capture that are going to taken it to the assessment scores etc etc but when we look at the wider campaign where we move perhaps away from pure training into awareness raising and knowledge delivering what works best and the two things that came out absolutely top were senior leadership messaging and live workplace event so they're completely ties in to what you've just been saying but if you can get in front of people and if you can have that senior leader input that's going to have the biggest impact and we know this and as developers and designers of online training I we know that when you're developing compliance training sometimes the behaviors you're trying to teach are very black and white and it's very easy to do online there are other times when you think I wish we could just have a conversation about this I wish we could get into a classroom or on a virtual session so a live workplace event doesn't necessarily need to be training it could be a forum it could be anything that gets people discussing and talking about compliance so I suppose the question I put out to Gary and Kim is really the insight we drew from the report is that anything the where you put people together is going to have the greatest effect what can you do outside of formal training to make that happen how can you make sure that your audience are hearing the same messages and the right messages from everyone from the CEO down to someone who's only been in the building for three months as a new hire and maybe I'll go to Gary on that one first yeah so I take a couple things we make a couple quick points and I'll turn it over to Kim I think you know clearly you have got to be doing this for the training program to the effect is that like your population has got to here be hearing not just you achieve compliance officer about your CEO or your head of global markets or whoever that is talk about these issues what we try to do and try to account for it again I'm going to build mode Kim is in you can still to our program I'm still building mine but what we try to do is account for in our kind of training and awareness program the town halls you know the quarterly meetings and having there be some reference kind of again conduct in compliance quite frankly though what we do tell our leaders when we when we do have you know kind of flight time with our leaders to talk about the execution of the program that it's also just really effective to have organic discussion so if somebody's having a team meeting and they mentioned the fact that Goldman Sachs just happened you know we're just in the newspaper because of an issue relating to confidential supervisory information and just a quick mention you know that can be a conversation that goes for one minute but it's organic it's real it's something that everybody can relate to and it's effective we're getting in the realm of things that are really hard to measure you know in terms of their effectiveness but there's no question I think you've got to have your senior leaders talking about this stuff yeah carry it we're on the same page with this one really starts at the top leveraging the town halls and other district type meetings or you know bigger staff meetings and one of the things that you know there's a campaign actually two campaigns on I'm helping build out right now and for both of them you know the discussion was how do we really get this message to people is it another course and I'm like no it's not of course it's a message that I would want to hear directly from my management my management team exactly and so why don't we put together talking points for managers to use when they're having their staff meeting you know just a couple of bullet points so that we have a little bit of control over the methods that's getting out right there's three things that you need to highlight and here's the wording for it you know so you can use those talking points and start the conversation with your teams or your direct report and I think tools like that or very helpful you know certainly if it's if it's a message from the very top or find that videos are are pretty impactful here people tend to like to look at videos of senior leaders chatting and most of them you know kind of come off as informal conversational but was definitely impactful but I think again just to echo what Gary said leveraging the senior meetings leveraging town halls staff meetings on down have a campaign structured that way and and give give leaders talking points to use so that they can they can all communicate the same message yeah thank you and Kim I think what you touched on at the end is something that we come back to often which is you have to make it easy for people to have those conversations if you if you if they feel threatened about the nature of the content or they feel overwhelmed by the task then those conversations don't happen and I think if you can make it easier for people by giving them speaking points by giving them structures for their conversations and by giving them directions as well so I mean we find it's very useful you say we would like you to have a 10-minute conversation or a 5-minute conversation or a 15-minute conversation you put a your ring scent is a little bit and people feel immediately more comfortable about the scope of what it is you're asking them to do right yeah great hundred percent right right so we're gonna have a very quick look at the last insight then I do want to spend five minutes if you can on questio
s and any questions we can't answer on this because of time we will respond with an email and so your question will be answered one way or another the last thing time was around contingent workers we found that 50% of organizations require consultants and contingent workers to take training as their full-time employee interestingly when you're within an organization you can sometimes feel that consultants and contingent workers are somehow other to the full full-time employees but that's not the way customers sit on the whole and it's certainly not the way regulators here so this is just an interesting one as soon as you start providing training for your consultant population or your contingent population it immediately becomes harder to track and score that training event because they're not inside your firewall sometimes as they're not able to access your LMS it becomes immediately more complex so I'd be interested to hear just just a couple of thoughts from maybe Goering Kim on how they manage this or any best practices they found Gary maybe we could start with you yeah so on this one you know it's kind of I pull my hair out over this issue to make it a long story short where we've landed is that if a consultant is either directly or indirectly exposed to the risk they have to take the training so the concept is if somebody's coming in to help us build a new risk model or to write a whistleblower policy I'm not sure why that person would have to take sanctions training but if that person is helping us clear alerts well given of course they would have to take the training everything you said about it's just hard to track you know kind of Kim and I are in positions where we just don't have a lot of leverage over this population to make them do it but we do hear our regulators always raising the issue with us but that's kind of where we landed if the consultant is directly or indirectly exposed to the risk like we make them take that that training yeah right yeah Carrie I think yeah I think we're sort of in the same place this is this is a tough one and one that we are trying to to always wrap our arms around so what Gary said definitely risk based approaches is necessary and the challenge is defining the audience right making sure that we've got the right people and then you've got people who have access to learning management system because they're in certain roles and then you've got folks that don't have access to learning management system so how do you train and track those folks and that turns out to be sort of a manual responsibility of the internal manager that's overseeing what they do and then it's you know when you when you're a huge organization its how you classify your employees so you may have someone who you know works in a capacity somewhere that has nothing to do with anything but they're on the list so how do you make sure that you don't force them to take the training and you know there's some other challenges jurisdictionally where you can let them have the training but it has to be a different version because you can't refer to them as an employee absolutely there are other things that you have to modify so it's a challenge it's always going to be I think a big nut to crack and you know we continue to do our best to just sort it out but I think this is a challenge for all of us and interesting Lee in other industries as well I mean when you when you get down to our Lee associates you have a whole whole new set of complexities to manage as well around whether you need to be paid for this stuff so I thank you both so much we've way through the insights we I think we've couple of people have asked during the session where we send a report out we absolutely will we'll send that to everyone who's on there on the phone today we did have a few questions come in I think Mark's going to pull those together now and just fire them out to either myself to Gary or Kim and just before we do that thank you Gary Thank You Kim so much for taking the time to be on this webinar felt like a really valuable 45 minutes I appreciate right here welcome eaten thanks Reverend yeah absolutely don't don't don't thank them just yet Matt we've actually had some pretty tough questions come in my project right yeah well there's some even a lot of interest I will try and synthesize where we've had multiple requests related to the to the same topic um maybe Kim you could open it up at this one how long do you allow an employee to take a typical piece of training so we've talked about how long the training should be but how many weeks do they have to get through their manager training each year before they start getting in trouble how many weeks or you mean if we launch a course what's to house due date um that's what I'm assuming the question is yeah generally only depends on it depends on the topic and that will vary on a global level it will vary on a you know sort of regional level but generally if the average is sixty days to complete a course some courses will be thirty but you know it varies depending upon the topic excellent same for us sixty days okay here's another one I'm at you might be able to chime in on this one with an instructional one how do you get confidence in your test down both from ensuring regulators are happy with the validity of the test and also your own senior management yeah so it can be tough conversation to have culturally to say that we're not necessarily going to show everybody all the content but I think the way you fulfilled training will help you to win that argument it comes down to what for training people is kind of common parlance it's about the learning objective for each piece and every question in an assessment needs to map to a specific piece of content that you have presented and then allow people to play with the practice as part of the course so the we don't build twenty minutes of training and then think okay let's pull ten call questions out of this twenty minutes actually everything that we present is the new piece of information has an accompanying practice section and then an accompanying assessment question and the reason we do that parts in fact that it gives good rigor to the training is we can then pull those assessment questions into a pretest so by doing so we can say if I if I answer question 2 incorrectly that means that piece of content 2 is the bit that I didn't get either because I didn't take it or I didn't understand if I didn't spend enough time in it and it becomes a lot easier to say well if you if you answer questions 2 4 & 7 incorrectly you need to review content 2 4 & 7 and we can start pushing people back into the funnel and just to look at the content they they didn't do so well on so it's really about how you structure your training in the first place every single assessment question has to be tying the bow tying the ribbon on the end of the learning objectives that you have already presented if you can't show that mapping you're never going to be able to win an argument around allowing people to test out fantastic well maybe move on to the next question just given the amount of time that we've got left Gary maybe you could lead us on this one what counts as instructor-led training today because of you know with the advent of virtual shuttle sessions with a manager how do you define an instructor-led training event today for that's a good question I am I'd say a couple things you know again Kim and I are in the math education business right so like when we're doing smaller groups kind of by definition we're probably less efficient in terms of educating numbers now I do believe that and he's kind of heard me before that I do believe a live well done is is is more effective I had like to get like 3,000 people trained live though you're leveraging telepresence you're sometimes you're broadcasting message it you're broadcasting the event to a larger population it's hard to train that many people live just kind of getting them in a room on auditorium one by one so it's a great question because when I say live I've kind of rushing li gotten to the place where you know telepresence captures live and certainly a broadcast council is like the key thing that I like to see is like that real life interaction with people and sometimes if you're witnessing that it's as effective and then sometimes if you can participate electronically it's as effective also I can't I'll turn it to you yeah I think you know I consider live training anything that involves whether people are in a physical room or in a virtual room and there are some interaction with an instructor we've done a pretty good job of doing both I think in certain areas jurisdictions we have got classroom training and then we've got virtual online training a lot of that is done as sort of the leadership level where it's you know the technology that allows a lot of interaction whether it's answering typing and in answering questions or responding or reacting to certain things and it seems to work very well the the challenge the other challenge with instructor-led training that's in the classroom as gary said is how do you get to everybody and we've successfully used the leadership structure to do that so that you know we we train the leadership team and then they're responsible for training others whether it's folks on your team or that they are guests and structures instructor topic and help kind of move the message and learning down in person but throughout the organization and you know smaller classroom setting so it is a challenge but it is very effective so those are just some of the other things that we've done fantastic some really great insights from everybody that I think at this stage we can say a big thank you to Gary and Kim it's been a super session and there were a couple of questions that we didn't get to so apologies on that but just before we ramp up as we stated we'll be providing a copy of the report to everyone that's participated but here might live to hand over to you maybe just to close us out and thank our presenters yes oh thank you Mark Thank You Kim thank you Gary and thanks so much for joining I think we've had probably a bigger attendance for Gary and Kim than we have for anything with on this ship makes me appreciate it again thanks everybody right again well
