Fax Observer Digital Sign with airSlate SignNow

Fax observer digital sign

thanks mike i appreciate that good morning everybody so uh well i had to struggle with a little bit of challenge to get online so uh we want to talk about uh the observable ability in pathology and look into possibilities how uh artificial intelligence could eventually help to medicate the effects and the impact of observer variability so um i as mike mentioned my name is hamid tizush i'm the director of kenya lab at the university of waterloo uh a member of water do ai institute and the faculty affiliate to uh vector institute so these are my disclosures disclosures so at the moment i have i'm doing some consulting activities for you on digital pathology so the motivation of course for us is that uh we know in us almost 12 million people experience some sort of diagnostic error every year the numbers of uh the number of fatalities are very different i have heard numbers anywhere between 25 000 to 250 000 uh death uh per year because of uh some sort of errors um 28 of diagnostic mistakes have some sort of life-threatening results and permanent disability and just as an example breast cancer misdiagnosis misdiagnosis costs almost four billion dollar a year so that's that's a serious uh uh problem and it could be that in the specific cases where we are working the numbers appear to be small but then when you scale it up and then you look at the look at the population you will see the impact well medical imaging is a large field with many many different branches from ct and ultrasound and mri to pat oct and so on microscopy is of course our focus and digital pathology in particular as virtual microscopy and there are millions and millions of images being captured every year for uh different purposes but mainly for diagnostic purposes so if we if we look at misdiagnosis as as one of the major problems that we have so there are different types of error generally so there could be a scanner that we fail to fixate on specific areas when we are looking at images and there could be a recognition error that we fail to detect abnormality so we we go over it we basically see it but we we do not recognize it but most problems happen in decision-making errors so when almost 50 of error is by incorrect interpretation of malignant benign as a benign malignant uh lesion or uh tissue so uh this is a this is a problem that we have spent a lot of time at the clinical community the uh research community the computer science community even uh here in the in an article from 2016 survey of 260 anatomic pathologists and 81 laboratory medical directors and what was interesting for me that they said have you been personally involved with a minor error so 71 of the anatomic pathologists say yes have you been personally involved in a serious error and almost half of them 47 uh say yes so that's um that's quite substantial and of course uh the uh the disclosure we don't have much transparency we have not look into procedures how we disclose and managers in in some aspect and as a as a general uh framework and platform so it's an issue so for example uh when you ask internets and surgeons uh uh have you in your disclosures have you used the word error so 71 of internet said yes 14 of surgeons say yes so or the first one would you definitely disclose an error to a patient so 65 percent of internet say yes 96 of surgeons say yes so it's uh it's uh it's uh it's something that depends on who you are what type of what at what end of the clinical um work are you active and then the responses and the numbers may be different so if you go back to let's say oncology and this these are these are markings of the prostate gland in a mri image by seven oncologists and uh it's unbelievable how much variability is in here and prostate gland is considered something really easy with its walnut shape and uh if we use this type of variability and for example in radiation oncology and of course we will not only miss the tumor part the red part here it will also impact the green part which is healthy tissue so it's it's uh the variability in oncology uh and specifically hearing radiation oncology is definitely a major uh concern and when we talk about delineation of regions of interest and images even a simple case like prostate has almost 18 18 percent of variability bladder 32 percent abdominal aorta 40 and you get two scary numbers of pulmonary nozzles up to 54 so uh and whenever you really want to delineate something on images that things get really uh this thing uh distinctly uh valuable and different from each other going back to pathology so there is a large number of uh reports uh that look at the observable variability here 20 uh 20 sections given to four observers and people are reporting capital values below 30 so 0.3 percent which is uh it's really uh is not really good numbers that you want to see in terms of our agreement but for some of the cases appear to be some consensus in in this report so that's a that's a relatively old one 1996. uh when you look again on for example bone marrow in bone marrow pathology and you're looking at the differences of subjective evaluations of three pathologies on the left we see uh how valuable they are so you don't even need to calculate any numbers to to appreciate uh the differences but also when we look at uh who's established independent factors you can still see uh you can still see the valuability so it's not it's not really about what are the criteria so the variability will be uh will be there and will be embedded anyway so uh another example so a uh uh looking at inter-observable variability for squamous and non-squamous non-a small cell lung cancer and so the percentage of agreement was anywhere in this society between 67 to 90 so and uh based on based on the primary uh analysis of the data the differentiation uh of non-squamous and squamous histology range from 77 to 94 so kappa values of 0.48 to 0.88 so um it it you you and you can see different similar numbers uh for different organs and uh uh subtypes if you're talking about cancer so uh another example is uh here uh for um for breast carcinoma 143 year old slide images and this was uh if i remember correctly this was specifically looking at virtual microscopy or digital pathology basically and uh six pathologists looking at it this is one of the things that has been missing because we know that variability does exist in the conventional microscopy but uh and we have a high concurrence in diagnosis between microscopy and digital pathology but there are not many studies who have looked at okay if i go digital will the variability increase or decrease so here among the things but pathologists we had a capital of 0.497 and this was about grading of uh breast carcinoma and when we talked about the grading of fear so great so grade two was uh we had a fair agreement and uh grade three they had a moderate agreement for grade one we had a good agreement so most likely intuitively the results can be understood so uh when things get really uh dominant and perhaps finding an agreement is uh much easier what is a scary to me as an as as a non-pathologist as a as a computer scientist is the intra observable ability so because you can intuitively understand that if uh experts sitting down and they have uh they are coming from different corners of anatomic pathology with different experiences and different specialities eventually they may disagree with each other but if if i have just one pathologist then the intuition of normal citizens like me as non-clinicians is that that that colleague that pathologist should always do the same things of course we know that that's not the case and uh in just to give you one example so uh three separate reviews of uh agreement of each individual with himself or his himself uh herself was moderate so we had we had uh agreement uh around fifty percent so capital values from zero point three three to 0.75 so which means if you give the same case to the same pathologist you would get different you would get different results and of course we know that again from literature but this is just uh uh is is just to uh to point out that it's not about that different pathologists disagree with each other so the valuability is something deeper than that that people disagree because they have different level of knowledge so well variability is probably the source of almost old problems we have uh so that and the question is what's going on in medical imaging well we have misdiagnosis we definitely have and then beside of that because of that and depending on that they have also inaccurate and improper treatment plan so uh what is the reason for probability well that's that's a very difficult question some of the reasons could be well the information is imperfect uh the imaging is imperfect anatomy uh is a imperfect in terms of there is no uh clear boundaries or the complexity of diseases and how they manifest themselves in in the shape of tissue is definitely not something that you cannot draw yes and no lines in many many cases and of course the human perception visual perception is inherently subjective so um among others this is this this list by no means is a complete list the reason of uh variability it it's it's a problem that probably and some of the papers even suggest that just meet you make your peace with it there is valuability accept it but just account for it and be prepared to deal with it i'm not sure i want to accept it so i i i think we can do something about it so um what are the consequences of vulnerability well we get based on that error or variability uh so you uh and if there is a wrong diagnosis and there is the wrong planning treatment of the patient or non-treatment of the patient or side effects for the patient could be prolonged treatment for the patient it could be reduced patient throughput for the clinic and hospital it could be financial burdens for the health care system and of course there could be legal ramifications depending on what type of healthcare system you are uh talking about everybody's talking about precision medicine and most of the time people mean uh to predict what type of treatment protocol are likely to succeed for a specific patient depending on various patient attributes and treatment context there is no question that if you don't have the right diagnosis the projected or the estimated treatment won't be correct either so that they go hand in hand i cannot really separate a diagnosis and treatment in my heart as a non-clinician okay what can i do there have been a lot of buzz uh about about ai we have supervised techniques we have unsupervised techniques we have weakly supervised techniques it could be algorithmic and topological anthropological or the so-called deep networks uh unsupervised ais are mainly uh clustering and search matching and deeply supervised is about interaction being online and interacting with human experts in order to learn so supervised you need a lot of labeled data so and then you talk about labeled data that somebody gives a diagnosis will delineate the the part of the image that uh are of interest then of course you will have the variability in it so any any any ai solution that has been trained with labeled data who labeled the data so did you account for observable variability yes or no most of the time i don't see it in the literature so weekly supervised you don't have label data but you give some sort of feedback reward and punishment to the agent to do its job and unsupervised that there is no teacher there is no reward and punishment you just operate the techniques operate on raw data you just give the images and the reports and the software twice tries to figure something out so uh artificial intelligence is a big field and uh machine learning is a subset of that uh an artificial neural network cnns are a smaller part of machine learning uh support vector machine that the classifier is a part of machine learning decision trees and export systems have been uh have been uh going around for quite some time random forests are relatively a new development of machine uh of decision trees natural language processing and rp is a has become a big part of an uh of um ai and sophisticated system like birth and bio birth have uh have been emerging in the past two two years and uh older techniques like fuzzy systems and metal heuristics like evolutionary optimization have also been around so what what what is uh quite successful in the past four or five years or the deep is the deep learning which is a small part of uh artificial neural networks uh and uh this is where where we get uh and hear about really impressive results okay so what is the ultimate solution for observable variability that's that's a tough question to ask and everything i say is not it's not really a solution that we can say we can use it tomorrow but it's based on everything we know something that we should we should go in that direction something that we should look at it as a potential solution so how can it can can ai remove observer variability so if i have an image and i classify it would that solve my problem because when i classify the image i most of the time i get yes i know is it long i don't know carcinoma yes or no or i'm after the grading i want to great uh i know what it is i just want to grade it and with the classification comes some sort of confidence or likelihood 96 so i i classify this as a squamous cell carcinoma with 96 uh confidence would that help me to uh to uh um get rid of observer variability what if you get all pathologists to accept this output then yes uh but is that possible is that possible that would that would imply a full automation which means so we just give the task we take the task from the pathologist and we give it to ai and whatever ai gives us we accept okay then of course the variability is removed because uh these techniques are quite uh consistent in the way that they do things but how likely is it that we accept full automation from today's perspective not very likely because we cannot understand this decision so we won't accept them and there is not there's not much perspective in short term that the classifications which are the most successful deep learning techniques there are most successful ai solutions are classifiers so and they have a lot of value you're not dismissing their value they're just asking can beside the value that they provide can they get rid of observer variability i would say no from today's perspective we can generate fake images synthetic images can that uh remove the variability i'm not sure because they are they create additional information that you have to basically analyze if what happens if you segment the image what happens if you find and delineate automatically using ai why that's great that's quantification that's the fundamentally the same thing as classification because you classify pixels but then again it would help to remove the variability if everybody accepts the result so if everybo and guaranteed if those segmentation techniques and delineations have been uh trained with uh label data coming from a few pathologists then the likelihood is is really large that the the result won't be accepted as as as a consensus what about search which is my favorite so what about search you give me an image and i give you some a set of similar images can that solve the problem of variability well okay so if i i'm looking at an image and i have a large archive of histopathology images and they are indexed so which means i can i can i can search in that archive i can send my query and say what is that and some sort of smart algorithm can search in that archive and send me back similar cases and say well yes we could find some similar cases and with the similar cases calm some metadata the reports the outcomes everything comes back and then i can look at it well this is not something new actually because the pathology consultations then the classes that we use are basically image search so when we consult each other when pathologists consult each other so fundamentally you are doing an image search in your mind we don't in your head in your brain we don't know how we do it but we do it and when we look in atlas it becomes more explicit if i look at if i grab one of the w shows uh tumor classification blue books and i go to the pages and i try to find the case that's similar to what i see under the microscope on the orange on the screen fundamentally i'm doing image search in my brain so we are doing it but we are not doing it with computers so uh and we know the benefit when we consult and when we look at the addresses we know the benefit but it's cumbersome is time consuming is not efficient so if if he had the possibility to search if he could send an entire biopsy sample and say okay can you find another patient similar to my patient if it was possible to send a small part of the tissue if i'm looking at the detail and i'm looking at really at four years and i am really interested in uh minute uh distinctions that i'm looking at or if i could select a region of interest in whole slide image and send that to the search engine would that help would that help to uh to search would that be possible to do that well if i'm looking at an image and let's say i have some doubt and i send it to an image search engine and say okay have you had a case like this so this is the consultation so that's what we are called virtual peer review and then the search engine goes and finds other biopsy samples that are similar to my patient and of course then the data comes with it the reports and the outcomes and the treatments and everything else and if there's any uh molecular data all comes back of course all those information um the the pathologist who's asking that is one person and the other information is coming from other pathologists who have already looked at other cases evidently diagnosed cases and they are in the system and you're looking at them so that's what that's what we call it virtual peer review so you basically tap into the knowledge and wisdom of your other colleagues and yourself in the hospital in the clinic and uh what comes is not just an yes or no it comes similar cases and yeah i have seen a case like that and it was popularity target carcinoma so which means what if i have a corey whole slide image as a pathologist and i send that to a search engine and we get multiple cases and with the report so basically we could build a computational consensus and the more we retrieve the more we find the easier it becomes to find consensus so then the magic is okay give me access to a large archive and the more in the more i have to at my disposal the more long cases i have the easier it should be to find cases to remove the uh uh the squamous uh case uh that caused variability that i mentioned at the beginning we we cannot really put some value on top of search unless we talk about natural language processing and the reports and then all the metadata that we have so most of the reports we have at the moment are unstructured so the pathologist just sits down and write the reports depending on the practices in different hospitals and clinics but they are also moving in towards structural reports synaptic reports and if if synaptics reports emerge and are widely used that makes the job of the computers and for search a lot easier so natural language processing can help to categorize reports and notes it can auto generate reports yes it can at the moment is very primitive but down the road it can be done quite sophisticated again provided that we have access to a large number of uh uh good reports which is not a given uh uh in the short term so there's a lot of a lot of obstacles to get there and conversational ai when the pathologist can just talk to an ai agent and ask questions to clarify something so uh when we look at the pathology reports there's there's from simple things to sophisticated things there's a lot of things we can do with the reports we can get a uh significant keywords out we can recognize topics we can highlight those things we can summarize it such that uh when we bring back the search results we can also highlight those keywords in the corresponding uh reports of the matched cases such that the comparison and decision making becomes uh easier and more efficient for the pathologist many people have started to not just look uh uh at whole slide images and their annotations for training some sort of ai techniques here a visual dictionary approach but also as you can see on the left also uh using reports diagnostic reports from the past cases as training data so you put in both images and reports such that the ai has a better chance to distinguish cases from each other and then you go online and use this system of course there is no report for the new patient for the new patient you just get the biopsy sample the whole slide image and then you do whatever you need to do uh is it is it for example reasons for prediction in this case for uh for prostate something really interesting is clinical report generation is this is one example uh on the left you see that the actual report is saying the nuclei are severely and the first sentence in the green text below it which is generated by the computer it says the nuclei are severely polymorphic of course it's not always that easy uh it's uh the research community as quiet at the beginning i would say mainly not because of the technology again because the lack of access to large uh number of data and getting one million reports is not easy so uh if there are clinical reports and they are in hospital uh it's not uh to my knowledge nobody has published any paper uh with uh with a large number of uh clinical reports uh evidently diagnosed cases but we have initial investigations that show this is possible we can we can auto generate reports and if i have access to match cases we can basically auto generate reports for them for that for an unseen case for a new patient so which means what which means if you give me the query and i go in and find the top three it could be top five top ten top hundred similar cases and i bring back those reports using natural language processing we could also put together the green report here and say not only just say this is popular return with carcinoma but also provide more information through uh a to a repository of nlp techniques and provide editable synaptic reports to the pathologist again the pathologist has to stay in loop the pathologist cannot go anywhere the pathologist has to there and look at the data and say yeah okay so it could be also upon request so we we don't want to bias what how the pathologist is working and if systems like this get established and then we can talk about and work toward establishing guidelines for using them but at the uh in the short term we have to look at some cases and see what would be the effect of using systems like this specifically matching similarities and image search on reducing eventually reducing the observable variability so there's a long way to go but uh but things are quiet uh uh quite interesting and quiet uh um uh um quite exciting at this stage so i will uh i will stop here and see whether we have any questions my thanks there to hamid for his talk a couple of questions how many have come in on the uh on the right-hand side um first of all um aston has asked them how are the rapid changes in terminology and the diagnostic techniques incorporated niftp versus ptc uh gliomas in general so you mean how did that um if are you in the um you use your hammock if you can see them yourself are you in the plenary stage chat and you'll see the questions coming in on the right hand side so let me see i have to of course yeah yeah so okay so the chat okay so um the question i asked there was from aston powers so it's in the chat box yeah how are the rapid changes in terminology and diagnostic techniques incorporated niftp versus ptc in general well i i have to generally say that at the moment uh nobody has made it to the clinic so that's a tough question for me because we still have to work i would say at least one open years to get something substantial to the clinics before we can answer this type of questions so at the moment we have a lot of stuff on the research side there is a lot many many activities and initiatives are going on to make sure that we also uh look at the user acceptance and the regular return side of things uh fta and uh for us in canada health canada to to bring the technology so uh and make it available and then we can answer questions like that at the moment you can only boost on the research results and the the validations on mainly non-clinical data so research data uh you can say that it will bring a lot of changes but what type of changes how would that be i'm really i'm really hesitant to make any prediction in that regard we know that we can change a lot so when i say v i mean the community at large pathologists computer scientists are specialists policymakers administrators all together uh we can we can bring about a lot of changes but we have not done it yet so there is no major uh diagnostic system deployed in pathology yet to my knowledge that can be used on a daily basis for making clinical decisions so we have to wait for that um everything i say comes from the research side of things all of them are encouraging but again we don't have hard clinical uh uh data to back things up which i'm not worried about it but but we don't have them yet sure and i realized i've missed a question that was asked earlier and it was from stanley cohen he asked them are you defining benign error in terms of difference from a subsequent expert or consensus diagnosis or based on actual patient outcome well i ideally remember actual patient that would be the good standard so but at the moment and when you talk about research we uh when you happen you rely on uh reports then you have to make sure that you have evidently diagnosed cases which is again based on outcome again i do not know any major uh major tests and major validation that has done that so anything that we have is heavily subject or ability anything we have is a few pathologists who have spent a lot of their time and energy and knowledge to enable uh a research study to look at the potentials of ai for diagnosis but which is fantastic but it doesn't solve our problem in long term because we're looking if if we don't do that we will drag the variability with us even with ai if the images and the reports are coming are coming from a few or even worst case scenario from one pathologist we cannot because ai will learn our biases so uh and uh uh ideally it should be based on outcome which is the absolute core standard we did this and that was the result so come back in the chain and make any adjustment that we need there is also a follow-up question to that and apologies if you've already uh tackled some of this within that but thomas wesley has asked related the excellent question above in the measurement of the intra inter and intra disagreements has anyone ever measured the outcome impact eg caught somewhere else in the workflow by other clinical indicators etc and how would you measure it tough question not to my knowledge but i may not be the right uh person uh to answer this question i'm i'm sure many many colleagues from the pathology side have much more in-depth uh uh overview over the reports on observable ability not to my knowledge and interesting is also is when we look at the inter-observability this is purely for research because in the practice we do not have that luxury we do not have the luxury to bring in three pathologists in the room to make a diagnosis intra-observability would be much more interesting because this is the daily practice this is the daily uh work that every pathologist in his her own room and alone with struggling and fighting uh answering the question alone so inter-observability is much more uh pervasive and much more uh a realistic case to look at it i don't know no i i i don't know that so uh not to my knowledge okay you always had lots of uh people mentioning that it was a tremendous early timely and insightful talk but there was one other last question um regardless from zev he asked with regards to verbal and written natural language translation interpretation in ai considering the vast differences between responses from siri alexa and google intro and intro available variability and your thoughts on that sorry i didn't get the question um it was your thoughts on your thoughts on verbal and written natural language translation and interpretation ai uh consider the vast differences between responses from siri alexa and google and again into an intra variability and if you had any thoughts on that sure so generally the ai community which i consider myself a part of the ai community does claim uh that nlp probably is much more advanced than what we have for processing images from computer vision mainly because nlp didn't have the major computational uh challenges as working with images does so we have put a lot more energy into nlp and training and lp solutions have been historically a lot easier so the progress has been made but when you look at uh auto captioning of national images when you show me a photo that you captured in part and the computer out to captions as i said a dog playing in the park okay so um uh that's fine but this is not the sensitive case that i'm looking at uh renal cell carcinoma kidney so and there we know that we have a huge variability in the terminology especially because we have been working with unstructured data especially because pathologists come from different background from different school of medicine terminologies are different and so on so would ai be able to incorporate that that pervasive variability in the language in learning such that everything that you want to do with auto captioning auto generation of report conversational layout would it be possible to do that theoretically i like to think yes practically i see a huge burden and challenge as we have for images that at the moment we do not have that large a diverse archive of documents reports such that the ai can learn the diversity can ai learn the diversity yes it can uh and we know that we can counteract the bias and the bias comes when everything comes from a few a few sample few pathologies a few uh clinics and hospitals so we need and we don't the initiatives are missing the initiatives are missing that hospital major hospitals get together multiple hospitals may be supported by governmental agencies and create large enough diverse enough data set to enable that as long as we don't have that we won't be able to exploit ai potential to a counteract observability both in text and in final diagnosis for images excellent well hamid that is uh all the questions and i think that was an absolutely fantastic talk to kick us off for the second day of the conference so my thanks once again for coming on live and giving this talk thank you