Online Signature Legitimateness for Insurance Industry in European Union

How to eSign a document: online signature legitimateness for Insurance Industry in European Union

good afternoon everyone and welcome to this webinar on auditing a digital Insurance word it's a big pleasure on behalf of vcie to welcome you in this very sunny afternoon here in Brazil as you might see in my background to discuss a very hot topic not just for the business but even for our private life because it's about artificial intelligence and also the way internal audit can help the organization and can audit to different programs that are in place the topic is currently in discussion in the European Parliament few meters from here so that's another reason why it's a Hot Topic because as you might know Europe is developing a new regulation that should be defined at the latest at the end of this year and is called the artificial intelligence Act so uh for today we will have some logistic details that I would like to share with you we will use a slido and you can connect to slido on your mobile or on your PC or any other tool using the QR code that you see over here we will use slider for ports we will use slider for Q a and of course you will ask me and what about the CPE for the CPE we will send you an email after the webinar asking your details so that we can ask the different National Institute to issue the document and send it by email to you we have together with the insurance committee developed in the context of this webinar a very interesting new position paper about artificial intelligence we will present of course the the main highlights during the webinar today but for details this paper will be available on the website just after the webinar and we will also send you the link to all participants together with together with the recording of today webinar so I just have a very important and last thing to do which is to introduce you our next speaker so we have the privilege to welcome today amongst us Astrid langavette who is the chief audit executive at Acme she is also a member of the ecia insurance committee and she's based in the Netherlands so Astrid huyamidak and welcome thank you it is first we have also together with us Chiara zilliani who is head of group audit analytics at generally So based in Italy a ciao piara and welcome hi hello everyone good afternoon who is audit hat group technology and operation group audit at Zurich Insurance based in Zurich So based in Switzerland I think I must say good good and dark something like that to you Robert so welcome and last but not least we have Frank hardens who is senior I.T auditor at akimea so Frank huya medak and welcome and I would also like to take the opportunity not not just to welcome you for the webinars today but also to welcome you as members of the working group that have prepared written reviewed adjusted this really great paper and you were with some colleagues so I take the opportunity to welcome the colleagues from akimea generali and Zurich insurance for the heart hard job big job and for the great paper so thank you to all of you for your support and now as to it I will let you as the facilitator the floor is yours thank you very much pascala for this nice introduction and yeah as you mentioned we we started last year already with this paper as part of our overarching theme we have uh have uh have developed within the insurance committee that the theme is let's say auditing a digital Insurance world uh and we picked up the topic on artificial intelligence and we started the topic in May without uh in September last year actually with our first brainstorm but we could not have anticipated that there will be such a big development over the last uh let's say half year we recently finished this this topic but it's uh it's all over the news let's say so it was a good choice uh and why uh auditing artificial intelligence well uh that's not uh that's an easy question I think uh to uh to answer because it's uh it's it's a it's a very hot topic let's say um and especially within the insurance company because um uh we think that uh in the current market context the the insurance company will uh the way they will operate will fundamentally change because of uh artificial intelligence uh the use of data and a algorithm is growing uh really fast and the idea it is actually expected to be a real key success picture uh key success factor also for companies to be uh to be ahead of the market uh artificial intelligence is of course already around us for for uh many years but uh because of let's say the more powerful hardware and and actually the Big Data uh which is available currently uh it's it's uh it makes a big step uh and and especially in the insurance business where we have huge quantities of data all over our insurance value chain uh this provides tremendous opportunities uh for example for for further automating our processes but also for the development of new and and more customer-centric products but also in our assessment of insurance risk so uh there are a lot of opportunities our survey showed that that the application currently is now centered around more the customer facing processes and pricing and called Blinds and uh not as much yet uh in in the real core processes such as claims handling and policy Administration but we expect that as always with new technologies that will will speed up currently there's a little bit of hesitance because probably uh we think that companies uh think it's it's it's uh well might be dangerous to to have no human intervention so that's probably the case why it's not in our core processes well that's a little bit my uh introduction um of course all these new developments uh influence the the risk profile of insurance companies and that is where we have been very busy on writing a paper on where the internal audit function should add value in this in this topic um and I would like to start with with our panelists uh with with a question actually uh what was the key challenge in your organization related to the implementation of our artificial intelligence and maybe Kiara as we said the ladies first maybe you can start with pleasure so um the key challenge also based on on my company experience I think a big challenge is the definition of a of an internal regulation that frames and provides discipline to AI application development companies must Define appropriate internal policies internal operational guidelines to um to harmonize the the organizational models and implementation of AI applications what we saw is that often AI model selection is addressed without a standard approach but mostly based I would say on a case-by-case analysis the point is that without an adequate internal regulatory framework all the requirements about interpretability transparency fairness and so on might become weaker and in the end the the ultimate risk is that the the development of AI application is in the end maybe of inadequate quality and afterward it increases the complexity in the maintenance of those applications that I would say is that is one of the biggest challenge yeah it introduces more complexity yes okay say something about the experience in your organization within Zurich yeah our organization started already a few years ago with an AI working group um defining policies defining governance um and what we always saw is regulation is often lagging um new developments so on one side that that was one of the challenge in a very changing regulatory environment to um develop governance and develop some processes and then on the other side you see huge changes also in the industry and the technology applied so this year it's it's the large language models and you don't know what we do will be next year so there's so much technology advances um and putting something in place which which is enough um specific but still agnostic of of Technology um and to be adaptable I think that was one of the challenge to um govern AI um but um I think also with our paper we will help um giving smaller but also larger companies to give some insights how they can do that yeah thank you Robert uh I'm Frank can you say something about the key challenges uh yeah I think to add to the uh importance of governance policies and strategies Etc around AI I think it's also good to um stress the importance of a consistent and um uh of a consistent development process throughout the entire organization across teams to make sure that there's no Q person risk in there to fully rely on data scientists that built the AI algorithm and I think that's important to make sure that that's been done consistently throughout the entire organization to make sure that the entire organization remains within their risk appetite um and also to make sure that business is really involved in all the key elements within that development cycle to make sure that a sound discussion is uh is is done on a for example the decision of a model which algorithm should we use which how should we train the algorithm that really also helps to explain the outcomes of the algorithm to make it a bit more explainable to increase the transparency throughout the organization I think that's also worthwhile mentioning okay yeah very clear longer of course we as a working group are very proud of our paper and and think it will at the values Robert was saying to uh to also uh well the bigger but also the smaller companies because uh it's as important for smaller companies because reputational risk is the same in a small or a big company we have uh set up the paper in in various parts and we would like now to explain a little bit about those parts so maybe Roberts it would be nice first because that's uh always a tricky one when we're talking about AI what is actually AI what is the definition what are we talking about so to set a little bit that that base maybe you can elaborate a little bit on that yeah there's there's a lot of definition on AI out and you can take different approaches to to define AI but when it comes to Insurance um regulation and insurance industry probably need to follow model has the definition of the regulators and uh for for our paper we we use the generally accepted definition uh of the oacd which is which is very wide but it's intentionally right um to ensure that no medium or high risk implementation of I AI are falling through the cracks and uh um when you read it out it sounds very simple it says machine based system that is capable of influencing the environment by producing an output so that's the simplest term um you you can get and there's there's only later in the definition there's some links to machine learning and advanced technology but already a very simple decision based system and can be biased and can be biased by very simple algorithms um by by a decision tree for example so in the sense of the regulator and also how we try to reflect with our paper and we kept the decision the definition um abroad aligned to the regulators and to ensure that all the relevant medium or high risk systems are in in that deficient definition category okay thank you yeah and it's still going on in regulation so it won't be uh well it won't be stable probably but this is I think a good uh starting point uh um maybe Frank uh Pascal was already mentioning uh we have the AI act in the European Parliament now uh in progress uh and so already on the first in in the first stage of development can you say something about the act in in the in a few a few words in a nutshell is that possible yeah sure sure um since we're talking about AI I thought why not ask a very familiar uh generative AI solution to help me on that so I asked that chat both to give me a two sentence description on what is the AIX and it came back and I quote it is a legislative proposal aimed at regulating AI systems within the you it seeks to establish a balance between Innovation and safe and trustworthy AI systems now you can argue about about the accuracy of that definition uh but it does touch upon that balance that is being sold between Innovation and safe and trustworthy AI systems and I think that's really reflected in the uh AI act with the four risk categories that are defined there it's basically uh categorize AI systems in four buckets for categories with the first being the uh unacceptable risk risk application of Airline and basically that is an AI system that is a clear threat to uh safety livelihoods and rights of people it's forbidden and cannot be put into the EU Market simple as that an example of such a unacceptable risk system is um uh social scoring by governance it's not not allowed to use human behavior to give benefits or even have negative consequences or on whatever second category is the high risk AI application and that's basically the uh the category of AI system that has the most strict requirements uh uh but is still allowed um examples of value risk AI assistance are um systems that can be used in a recruitment process and selecting a possible candidates for a job position that's not that's that is allowed but on the restrict requirements um those requirements come in quite a different uh a quite a huge set of specific elements that need to be met for example around governance and risk management systems for example around transparency to users and human oversight but also around technical documentation and how to measure the system's accuracy for business uh cyber security so it's a very extensive list of requirements that needs to be met before the system can be put into the market and then finally there is the limited risk and a minimal risk category um that is basically the rest of the AI systems um some additional requirements may be needed for example a jetbots for chatbots it must be very clear that people are speaking with chatbots but um yeah it's definitely not as extensive as the high risk AI category um well that basically means that a a a lot of work is to be done yeah you have to be make sure that we comply with all those additional requirements a way that is companies and governance that I want to use AI systems in the European market um it requires a a sort of a CE marking before you can put that system into place including a a check a self-assessment or something like that to make sure that you comply with all the the requirements and a non-compliance comes with um quite excessive fines can come with excessive points like six percent of a global annual turnover so that's can be quite a lot of money yeah absolutely so work is to be done uh uh the question remains of course when should that work be done when should we be prepared for that a new AI act that is coming and that unfortunately is a question that I cannot answer with a concrete date but the European committee is aiming for early 2024 as Pascal mentioned earlier um yeah but that all needs to be put into the perspective of the European political Arena um I think this week is an important uh a time uh there is this uh um uh draft location a mandate that needs to be formally accepted by all by all um by the old Parliament and after that uh the the real negation negotiation for the Final Act start and um yeah I think last week there was this news item that you uh uh still new amendments were being tabled and so that makes that decision that votes this week really important and a really a key decision moment in um the next step of the AI act and also of course then um are we uh when can we expect the ACT to be finalized uh one thing is for sure whether it's is uh early uh 2024 whether it's earlier about a place later the impact can be um that can be huge okay thank you Frank so uh that's very important also I understand to have the right risk classification then for your uh AI applications that will be here it really starts with uh knowing all the AI systems that you have that you put on the markets and risk assessing them uh that's the first key step okay thank you Kiara uh yes uh you've made a tremendous effort in writing down uh issues around all the impacts of AI for example on strategy and governance and legal and compliance can you say something about that yes so about mentioning all the impacts that AI will have is is a is a is a material task so let me share a couple of uh of example a a big impact first of all is obviously on the operations management because the operations are dramatically changing thanks to the technological innovation connected to the artificial intelligence what we saw as you were mentioning Astrid is that in the insurance sector at the moment the areas most backed by AI are the processes customer facing for example the customer service where the aim is to improve customer satisfaction and at the same time reducing costs there is a less implementation in the administrative areas for example financial reporting or policy Administration in these areas what we saw is that companies are probably still a bit reluctant to rely on AI and consider that the human intervention is still significantly needed probably it's it's just a matter of time and in in the future we will see that also these processes will migrate to the towards AI system the point is that with these new technologies new risks also come up and I'm referring to risk related to to compliance to security uh data protection ethics the what risk man where risk management should start from is to make sure that the AI strategy is aligned to the business company to the business strategy of the company and to the company respect Capital specific policies and guidelines must provide guidance on the company's definition of AI on the of the on the kind of AI applications that are allowed and the ones that are not allowed provide guidance also on the extent of decision making that is allowed for AI systems so such a framework should provide transparency within the company about hey I should be used and now it should not be used because clear guidance helps the organization to find Opportunities to get the benefits of AI but at the same time remaining within the company strategy and the company risk appetite so first big impact I would say definitely on the operations management but another significant impact is the is the social one uh which is still difficult to predict and quantify but for sure will be huge considering the speed of development of artificial intelligence it's still not so clear what this impact will be precisely for sure AI will have a positive impact in areas such as Healthcare transports communication uh nevertheless there are some potential risks connected to um to the fact for example that AI could perpetuate and amplify existing Prejudice and social inequalities especially when using historical training data think for example of risks such as bias privacy violations unethical use use of s all these risks are at the origin of the increase in attention by the regulatory bodies because the question is how can AI be regulated to prevent it causes harm to people but why still unlocking the potential because there is a huge potential in the use of artificial intelligence which which should be unlocked at the moment AI is not unregulated there are already data protection and privacy laws which already set some some boundaries however as Frank just explained there are new more prescriptive AI laws that are that are under definition and beside this uh comprehensive regulation being developed there are already many principles developed for example EU has recently adopted a recommendation on the ethics of artificial intelligence which provides a set of principle and guidelines for developing and using AI in a responsible and trustworthy manner so countries regulators and Industry bodies all around the world are now rapidly moving uh forward with the legislation in uh in this field and the common denominator is always that they should address concern about transparency of machine decision making and uh and ethics I think very clear uh Kiara thank you very much I think we have Upon Our Pascal and that is a poll on uh yeah the current role you have as attendees of this webinar uh related to the assessment of artificial intelligence maybe you can provide an answer is that only advisory or insights okay none or do you already included in your risk assessment or maybe some of you already provide full assurance yeah I think we have the kind of results it's not really changing a lot anymore it's none stays around 70 that is uh there's quite a large number actually so uh but that's why it's very good that you attend this this webinar uh but also I see some of you who already have uh full Assurance audits on it so that is is good to see but the conclusion is a little bit that we have uh some step to take as as internal audit function within the insurance industry and that is also a little bit in line with the results of our survey uh isn't it Robert because well that's something we saw also in the survey and maybe you can see something how it is done within your organization foreign what we saw is that the larger insurance companies already had an approach and the smaller emit size they they often still has have no policy but also no audit activity over um AI plans or I have not performed anything they lost 24 months and in our organization everything what we're doing is aligned to our mandate so also in artificial intelligence um we are providing Assurance similar than over and on the writing process or over a claims management process and we for example select a few AI instances which which are either live or already or in development and we look at the governance over AI the risk assessment of second line and the controls implemented um weep example developed our own methodology which we call algorithm um review and testing and you see also in the appendix of our paper very very well outline methodology you can apply yourself so in a nutshell we really provide Assurance agreed with our boards and all the committees as as part of our audit mandate um similar within to any any other risk area we need to cover okay then I said well in general uh Kiara so also also in our case I mean the the the the role the the mission of internal auditing is stated in the endured policy and uh um what we do compare in in the area of artificial intelligence does not differ that much compared to what we do in in other areas so the mission of internal audit is to enhance and protect the value of the organization by providing risk space and objective Assurance advice and decide and decide this applies to artificial intelligence as to any other areas where we provide Assurance the internal audit function plans and can result the activities following our specific reported group audit methodology in addition we might carry out advisory activities for the board the Senior Management and other stakeholders with natural scope agreed to them for what relates to AI has said not really different in the case of AI internal audit in generally has played both roles on one side we have played a pure Assurance role we have been auditing AI governance and risk management with an audit program encompassing the seven key risk areas strategy and governance legal and compliance system development operations management security and data protection the human capital which is a very important factor and sustainability on the other end we have also played as advisor upon the request of the of the first line when the first line in general in general it developed their own smart Automation and artificial intelligence which framework they ask for advice to all the control functions and independent advice in order to build it more more properly of course we maintain now we would provide this advice wise maintaining our independence okay yeah thank you and Frank can you say something out about our company yeah I think that we follow a similar approach as uh giardos of course we try to give Assurance where possible but in this case for uh artificial intelligence we started we follow the maturity of the organization that means that we started with a uh with insights and advisory services on the standards the framework that was established around AI uh and we're now getting into the uh uh uh into our formal Assurance rule by selecting uh one two AI systems and a really audit uh if they comply with our internal and external requirements okay thank you yeah that's clear so we are speeding up uh and and following the speed of the organization uh sorry other two big surf okay um we have another poll I would like to have your input and that is uh related to the aspects you are currently uh including in your audit activities uh and you're assessing so we uh we pointed out a few aspects so the strategy the governance also the actually the things that also Kiara Frank and Robert were already mentioning foreign yeah I think the outcome is clear I think and which is very good I think that uh as I think also the panelists were mentioning it's very important to start with the right governance and to really have it aligned with your strategy so uh these two topics are we'll say uh at least having high percentages so that's very good um and yeah well these are also important topic we included uh in in the order program we we mentioned uh maybe we can uh discuss a little bit on that on on how actually do you audit those various components uh uh maybe you uh Kiara can start uh with a few of them uh yes strategy and governance which is very much voted yeah it's fine it's it's nice that it's on top of the list so you can start yeah yes yes sure so here the risk is that maybe weaknesses in the strategy definition or poor governance or AI initiatives might lead to failure in the implementation or around the performance compared to competitors in the market and this might be originated by several causes first of all lack of a Clear Vision about the risks and the opportunities of AI for the company a second poor setup or poor organizational commitments in the company for AI projects third Point third possible let's say root causes is the um maybe a lack of appropriate performance metrics to monitor the AI applications but most of all as already mentioned many times lack of defined and approved policies procedures operating models clarification I would say offers and responsibilities so what auditor should check is exactly this has an AI strategies been defined and documented and cascaded within the organization how are the AI initiatives communicated within the organization is there a monitoring of the accomplishment of the objectives in place and most of all is there a governance framework operating models being defined that that's really key questions that we should that we should ask when auditing this areas related to to strategy and and governance and another key areas I would I would mention is the one related to legal and compliance here the risk is that compliance breaches might lead to adverse publicity regulatory fines and in the end also reputational risk again this might be originated by several causes first of all obvious missing analysis or acknowledgment of legal and supervisory requirements including all the ones related to data access and data treatment but also especially considering the last development also missing awareness of the ethical aspects in the AI application we we made the example of the EU was recently adopted this recommendation of the ethics of artificial intelligence which which provides really a set of principle and guidelines for uh for using AI in a responsible Manner and some of these key principle relates to human dignity human agency and autonomy fairness democracy privacy and data protections so Auditors of this respect should really question themselves have all the relevant regulation and international standards first of all being identified and then embedded in the internal framework are all the AI initiatives prepare for compliance with the existing and upcoming regulation about gdpr for example all the upcoming EU AI Act but again about that specifically about data is the necessity of using personal data being assessed before using those those data how personal data will be used to train AI system and which is the purpose for for using them about ethics just to conclude did the company carry out a fundamental writing path assessment is it possible that the AI system could interfere with the user decision-making processing in an unintended way if we if we use a chatbot for example or another conversational agent is the human end user aware of that do they know that they are talking to a non-new manager these are type of very specific consideration that should be taken into account when auditing these areas in artificial intelligence just to give some example yes very clear yeah it's really a matter of as I understand you correctly to really understand your business what is going on and to be sure there's uh yeah there's a overview and uh and and you have a complete view of all the initiatives as an auditor also and uh and and to assess whether the company has that overview but also to have a good view and understanding of all the regulations and standards and and risks of course identified and that can be very impactful especially those ethical and social uh risks and so that's very clear and uh maybe from you can say something about let's say the more technical part of AI the in the performance uh technical performance yeah uh of course uh of course um how do we all the data that's exactly yeah well uh um I think Ikea already stressed the importance of good governance and also about the input data for the development and in my opinion it all comes down to a solid AI development cycle all checks and balances within the development cycle need to be there to make sure that the uh um uh the model is trained accurate to accurately that's uh the right model is being chosen uh that the organization really challenged themselves in how to measure the performance of the uh uh of the AI system uh you all want to see that embedded within the development cycle because in case of a um mistakes within the development cycle so you can immediately expect results of that AI system to be uh yeah at least less good you want that cycle to be up up the standard to make sure that the results are good um yeah the risk there with offer a bad development cycle is of course that uh um the poor performance uh poor a performance over time perhaps also because uh predictions might be good at this moment but you want to make sure that they're good over time as well um results may be a a a a a difficult to explain by the business uh you want to make sure that you make all the decisions within the development cycle to make sure that you can explain the outcomes of the model afterwards um yeah and if you don't meet those requirements of course there can be consequences like reputation loss Financial loss etc etc um I think the key elements are within that development cycle also around security that is because if the security of an AI system is not upstanded training data may be compromised maybe inaccurate maybe false end result to false results in the end or uh AI algorithm may be stolen in itself making it a loss of uh um uh intellectual property uh so also security within the development cycle should be addressed appropriately however if we want to a a audit that development cycle if we want to audit security within the development cycle um in essence there can be you can apply two approaches you can use the desktop approach in which you basically follow the development cycle use audit Trail within the development cycle to test if all the controls are in place to make sure that uh the right model is chosen to make sure that the right input data is chosen to make sure that appropriate testing is being performed on the outcomes etc etc uh that really helps in giving back your your feedback your uh uh uh uh um recommendations your findings on the Improvement on that development cycle on the other end is also a possibility to use a full review approach and that approach basically a a you as an auditor try to reproduce at least parts of the algorithm to make sure that the output or the algorithm is accurate so you uh uh uh reproduce parts of the algorithm to check if the results that you get from your basic model that is if they uh a match with results that are expected based on the Real Models performance um of course both techniques have a a H their benefits and their their advantages and their disadvantages but they can really help in a a good decision in which approach you choose can really help in making the right impact for your organization okay thank you Frank uh maybe Robert you would like to add anything to already these topics on the other program of our AI do we miss anything for example like human capital part yeah so the human capital can be a root cause for elements which are not working for example lack of training lack of understanding lack of knowledge and lack of um Workforce Development um Within These these new areas um so once I we can interpret that as root causes of of our findings but on the other side we can also audit these elements as being also a control on a more the softer side um to ensure that um AI is implemented the way that the company wants and it's aligned to the strategy and another topic is ESG we see a lot of ESG controls um also being developed in parallel to new er regulations and as soon as AI is used for any of these elements in the ESG space you have similar issues then with any AI for example decision investment impacting in in greenmash watching sanctions and for example Road procurement decisions and so we have a lot of areas which which are also overlapping with ESG relevant controls and we find that also in our section from another program perspective that he also consider these risks in addition yes thank you thank you very much we now do have time for uh some questions um I don't know whether our questions are Anonymous uh you already touched a little bit on that I think Robert the the impact of sustainability is on IA models you say there's a lot of uh things in common but uh it's also impacting sustainability on itself it's using a lot of electricity also that's an issue yeah great so um I mean all all computer power using the cloud and using extensive calculation uh engine producing uh carbon emission in its own and and you see for example the discussion on crypto and crypto mining um it's already using uh an average size consumption of of Belgium so we we expect the AI itself being um a contributor to EST but on the other side there's a lot of positives um environmental [Music] um Elements which could be triggered by the use of AI for example better model better predicting model on the natural disaster hurricane slots thoughts prevention then a lot of socio-economic impacts better medicine better training imagery these doctors medical design so there's a lot of areas which AI will will touch in the sustainability and you probably don't even know what what the consequences will be yeah we already mentioned at the start it can help us to assess as insurance that's very important to assess the risks especially the insured risks and uh and maybe even then prevent it which might be helpful for from a sustainability perspective uh we have one top question I see which is lagged a lot um and that has a little bit to do oh it's certain the second one now what Vivid examples do we have of using uh AI I think in internal Audits and can it achieve extra extraordinary results that Auditors can notice so will it replace our work I think that is not something we focused upon in our paper that might be an interesting topic for another one but we focus really on the application in in the business itself but is there anybody if you would like to comment on that do we already have real good examples of AI and all the things yeah yeah I'm happy to start and then my colleagues can chop in but um I think as as an interloaded function you should be using the same tools then the business are starting to use so and it also it's a great training ground for your data scientist if you have a data science team in your other team to start using machine learning start using natural language processing start using all these modern tools for audit cases and audit purposes and we we made actually great progress in using them and we have great outputs already as well and finding um elements we would not be able with any other technology so for example the network analysis in in procurement to find out the interrelationships between different vendors and different business units Based on transactional data we would not be able to do that with millions of data sets um so there's so many use cases you can apply in in the machine learning space to your audit questions and I really hope that yeah I feel fully agree on that Roble I think NLP is great to process this huge documents huge text to find the key element is in it to see if if the document in itself is relevant for your audits but also you can use data mining techniques algorithms to find those transactions that really stand out uh once what you that you probably would not find doing a random uh a sample selection of for example so I fully agree I think there's quite a few techniques that are really useful okay yeah and you think Kiana will also have some uh examples within your organization I think where you use it yeah something something I might add is that uh those techniques are useful not just to execute audits but also to enhance um our internal operations for example as Auditors we collect tons of data that are collected as in a repository basically in our audit Management systems but thanks to artificial intelligence Technique we can extract Insight from those datas something that a human could not do unless opening everything but thanks to AI we could extract import we could see Trends if there are patterns if there are trending topics within all these data collected in our in our systems okay okay thank you Kiara I think yeah we have 10 so we have to write another paper I think on this topic um when looking at it was the the second most popular question what approach shoots internal audits follow when auditing AI uh fighting Assurance actually on AI that's something you can I get I can I can take this one if you if you want um well the IIA says that internal audit should approach AI as it approaches everything so not in a very different way with let's say systematic and disciplined methods uh to evaluate and improve the effectiveness of the internal control systems related to to AI so the approach is not really different from the one that would be applied in any other area what changes are the competencies required to make this approach systematic and discipline first of all internal audit should recognize that new skill sets are required internal audit must have collectively a sufficient understanding of artificial intelligence and of how the organization is using it and of the risk that AI represents for the organization in alternative organization internal audit might also bringing from the external those uh those expertise required when performing an audit on on AI and also uh recognize that auditing AI does not only require knowledge and expertise about technical aspects of AI but also knowledge regarding data governance data quality ethics all this is is equally important and eternal audit you should consider all this yeah yeah and and as you mentioned before the the real business knowledge what is it really how is it really impacting business and and for example customer interfacing and really the business understanding I think we're already uh a little bit running out of time uh posca I would like to finalize this webinar with uh with with our it's a concluding remarks or takeaways uh uh maybe from you can start what is your key takeaway from uh from this webinar but also working on on the paper together yeah I think that the my key takeaway is to um really see a um AI Assurance essay integrated part of everything else that is already there it's not Assurance on AI in itself but elements like Model Management like uh software development like uh data data governance all those aspects most of it quite a lot of those aspects are already in place at the organization I think it's really good if companies to seem to try to seek um [Music] for real integration in everything that is already there in place I think that that will be my key takeaway okay what's your key takeaway even if I'm going to to repeat myself uh what we saw is that the organizational still relatively immature in respect of auditing digital Technologies but also in the case of more mature organizations uh sometimes the opportunity erased by the implementations of AI models are not always accompanied by an appropriate management of the related risks in particular there is still a lot to do in building a strong governance and the robustructure framework governing this this challenging opportunities I'm talking or referring to policies guidelines all what is needed to have an appropriate discipline of this of this new technology yes thank you yeah it's very clear uh and Roberts what is when you look back a little bit what is a key takeaway nobody should forget yeah it was interesting when we developed the paper um a lot of changes were only coming I said at the beginning with the large language models this year and there's still there's no consensus in in the society um how to deal with all these developments and but what is sure is a I will stay it will not go away and it will be used wherever it can be used to generate value and that's really the important thing um AI is not something good or something bad it's really how it's applied and how how it's um used for for a better good and I think internal audit can really play a role here that it is you sort of at the good and not for something which can harm Society or um individuals so that that that with that that would really conclude our learning that yeah yeah I I agree and maybe that's that's my key takeaway that that uh things are developing fast and you should say especially over the last couple of months the monster large language models but it's really impacting more and more uh the business and our environment and we as internal Auditors can provide a very important role in in providing insights and advice but also Assurance because it is very important to have trustworthy AI as as we always call it so that will be my concluding remark uh so Pascal I don't know whether you want to wrap up the the meeting yourself but I would like to thank everybody who's attending this webinar and of course uh thank my colleagues from groups and Kiara was wonderful to uh to to cooperate on on this paper and I hope everybody will read the paper uh which which lots of uh lots of interest and will uh and of course we expect it will be very helpful to uh to make a next step in in auditing artificial intelligence and and add value to your