Write Heterogenous Time with airSlate SignNow

Write heterogenous time

hello the short video is intended to run through some of the most important aspects to think about when you're writing multiple choice questions in order for them to be both effective in assessing the learners and that the results are going to be psychometrically sound writing good questions is definitely something that's a learned skill not instinctive the good news is is not that difficult to learn the basic rules to improve your question writing before you even start writing a question on topic this isn't the most important question is it really worth evaluating asking learners to recall esoteric facts and percentages of instance diseases etc is often not really worth asking so I don't waste your time asking questions that aren't really important in our daily practice I don't know if you're familiar with Bloom's taxonomy of learning objectives but we really want our learning objectives to be near the top of this pyramid so we want to be asking questions that test evaluation or synthesis or analysis of knowledge rather than just do they understand it and can they recall some fact and the American Board of Radiology aims to have a greater than 90% of its questions in this top four parts of Bloom's taxonomy each of these levels and Bloom's taxonomy has some verbs that are associated with it you can use in your questions or to think of your questions and you can easily search for this on the internet if you want to look at it further so let's look at how an individual question is broken down it was broken down into four parts the stem the leading statement the answer options and then the key answer and I'm going to go through these in turn so the stem is typically presented as a very short clinical case scenario for the exams the American Board of radiology and also for boards exams for national board of medical examiners for medical students keep is very short and this should be a linear present so for example a 50 year old man presents to the emergency room with the Domino pain a CT scan was performed what do I mean by linear presentation so first of all the sex in the age any relevant past medical history then the presenting signs and symptoms then what test was done obviously for us it's usually imaging procedure and then the lead-in question this is cognitively the easiest way for people to read a question and not skip important parts of the information so this being example of a nonlinear presentation of a question a CT scan was performed on a 32 year old man he presented to the emergency room with abdominal pain he has a long history of alcohol abuse and we try and avoid this kind of a stem the lydian is the question and this could it be either in the form of a question on open statements so a question would be what is the most likely diagnosis or an open statement is the most likely diagnosis is or which of the following is the most appropriate if you use one of those latter two types the open statement the answers should always complete the question so often it's easier to just put it in the form of a question to my mind so very typical lead-ins for us as images would be what is the most likely diagnosis what is the best interpretation what is the most likely cause of these findings what is the most appropriate cause of this clinical situation and so on if normal is one of your answers then you should use what is the best interpretation as opposed to what is the most likely diagnosis because normal is not really a diagnosis the distractors are the wrong answers usually have three or four wrong answers giving you a total of four or five total answer options it doesn't have to be that number there's no magic about that number but you certainly you want to try and avoid having less than four answers in total the only time you can really have three is if there's no other option for example it's less than equal to or more than or increase stays the same and decreased the wrong answers need to be plausible and that's really am you know that's really important don't just toss in something that clearly cannot possibly happen and they need to be very homogeneous which we'll talk about in a moment ideally want these distractors to range from you know not at all like you're extremely unlikely to less likely than the correct answer because that that mimics real life you know when we're looking at a CT scan and there's an abnormality there's a bunch of different things it could be some of them are not very likely and then there's one that's the most likely and you want that most likely one to be you know kind of a significant step ahead of the others in likelihood the distractors should all be similar in length and the format and they should be balanced so for example if you have two which use the word increased you want to have two which have the word decreased within them the key is the correct answer so we can identify this when you're writing a question with them and submitting it for exam board with an asterisks or written underneath it says key equals a this might be the only one correct answer or in some cases it's the most correct answer when the others are possible but significantly less likely and obviously this will depend on how you phrase your lead in some some general rules to question writing keep the stems and the distractors as short as possible don't put in unnecessary words and unnecessary information it's just going to mean that it's going to distract the test takers from the important information there it's really important that there is no ambiguity in the interpretation of this question and when you look at the question and for image based questions can they answer the question without even looking at the image you know it's not a good question if they can answer it without looking at the image here's an example of a question that if you're a smart test taker and you're used to looking at the clues in questions you can answer so this was a obviously an image-based question with odd sound which I'm not showing you you don't need the ultrasound because what do you look at you look at common words there's kidney kidney kidney this is kidney and kidney so we know that kidney is going to be in the answer then we look and we say okay the ston stone there's three instances of stone so I'm going to look for one for stone with it there's heterogeneous mass appears three times I'm going to look at the commonalities of this and I'm going to come out and say all right the most occurrences of these common elements appears in D and guess what this is the right answer for this particular question so with an image-based question you want them to have to look at the image to be able to get the right answer there's a second important test and this is for both image and non image based questions which is can it pass the cover test in other words if you cover up the answer options not the image but you cover up the answer options can they answer that question if they were smart knowledgeable test takers so for example which is which of the following is the most common type of breast cancer you shouldn't need to see the options to be able to answer that question and I'm going to show you some examples of questions that fail the cover test later on we want to try and make the distractors as short as possible so if there's common terms within the answers move them into the lead-in so here you know this is common term cerebral artery in each of the answers so what can we do so we can take out cerebral arteries and move it into the stem and setting instead of saying which of the following vessels has been included we can see which are the following cerebral arteries has been included and hence make the answers shorter over the years a variety of different answer formats have been used but now these been whittled down to really very few and they're really variations of only one answer format and that's the single positive answer so an example of a single positive answer would be a leading which says which the following is the most likely cause of this abnormality so answers are all positive not negative there are a couple of other somewhat more complicated answer formats which are accepted but really they're all based on this I'm not going to go into them in this short talk I do however want to show you some of the invalid formats which people often write questions in I'm not going to talk about why these invalid if anybody wants more information about why psychometrically these don't score well in exams then I would suggest you read either the American border radiology or the nbme exam writing guide you should not write items have negative stems so which of the following is the least likely to present as an abdominal mass or all of the following are true accept and you know why these bad because many exam takers will miss these key words here they know the information but they will get the answer wrong because they look for the which of the following is the most likely to present as an abdominal mass or they miss out the accept and to see all of the following are true and so on you can however use which of the following are contraindicated that's considered to be okay similarly you want to avoid having negative terms within the answer stem so which of the following statements regarding breast cancer is true and the answer stem here says it is not the most common cause of cancer in women how do people read that they read it as it is the most common cause of cancer and women you should not use questions that have multiple correct answers which are really multiple true/false which of the following diseases might cause us finding select one or more so you should have single positive answers similarly true/false questions such as identify which of the following answers are true statements these have been used in the past they're no longer used in national exams and if we're they're not used in national exams we shouldn't be using them in our test writing for our learners don't use all of the above a B seedy and all of the above we all know this you know we've we've taken many of these tests and we know that if we see that a and B are true then we just skip to eat we don't even have to think about C and D an old exam type that used to be used or the ones that were a B and C and then a and B a B and C and so on again don't use these I just want to run through a few of the common mistakes that I see I've done a an awful lot of exam item editing and these are ones that come up again and again don't use inconsistent units the unit's here and you can see I've got millimeters centimeters micrometers you want to have the same units for each of the answers and not only that but you want to put them in numerically increasing order of size don't have numerical values overlap so you know in this particular case if the learner thinks that the answer is 5% is it going to choose answer a or is it going to choose answer B so make sure it's something like 0 to 5 percent 6 to 10 you know 11 to 15 and so on keep the answers homogeneous they need to be focus so therefore they may should be the answer should be all diseases or they should be all artifacts or they should be all numeric or they should be all physical findings and so on rather than a mixture the others and I'll show you example in a minute and it and in this this is going to help you pass the cover test so here's an example of unfocused or heterogeneous answers if you look at these here this is a statement about the disease as is this this is a radiographic finding this is a radiation dose these are very heterogeneous answers and not only that but this does not come pass a cover test you know if you covered up these answers you cannot answer this question however brilliant you are so this is an example of focused answers which of the following is the most common type of breast cancer so these are all pathological diagnosed sister or fill into one homogeneous category and not only that but it passes the cava test if I covered up these answers you should know the answer if you knew the information try and keep all of the answers the same length we have a tendency to might write a much longer answer for the correct option than we do for the incorrect options because we spend a lot more time thinking about that and then we kind of toss in the others so that's part of cluing we're cluing the readers of this exam that this is the correct answer here and the others which are much shorter are the incorrect ones and you can falsely clue this as well so you know people who are good test takers will look either consciously or unconsciously for that longer answer and can be fooled if you're not keeping them in the same length and you can usually rewrite ones to be pretty much the same length fairly easily if you can't then have you know perhaps two that are longer and two that are shorter or three they're longer and two other shorter and so on if you're using lead ends which are not in the form of a question but in the fold completion of a sentence you need to make sure they're grammatically correct so the most likely cause of this abnormality is metastatic renal cancer that works myeloma works fiber dysplasia works but the most likely cause of this abnormality is needs correlation with radiography does not work and immediately Clues the exam taker that that is an incorrect answer be very careful with relative and absolute terms so relative terms such as sometimes occasionally usually they're extremely imprecise and you know there's many studies that have looked at one person's interpretation of these terms compared to another person's and the correlation coefficients are extremely low some people think that sometimes is 10 percent at the time and other people think that sometimes it's 70 percent at the time so avoid using these and simile terms such as always never and must be absolute terms we all know that that never happens I mean there's nothing that's never and nothing that's always so we will immediately exclude those as being viable options there are some subtle aspects of cluing that you may not be aware of although I'm betting that you have actually subconsciously use them when you've taken your own exams such as repeating one or more word either in the stem and the answers or in more than to answer options kind of like I showed you with that case of the renal and liver masses previously so here's an example here a 60 odd man arrives in the emergency room in ventricular fibrillation after resuscitation a cardiac CT scan is performed what is the most likely diagnosis and here is the word ventricular repeated again in this particular patient I was I'm not showing you the image but that Clues you to look for that ventricular may be more likely same history here but in this case the word atrial is repeated in two of the answers and that's going to clue you that it's either atrial like home or H or myxoma is the correct answer what are some ways of getting around this well you can repeat the same terms in what's called match pairs so this is ventricular twice here and there's atrial twice there is no clue in this as to whether we're going to point towards ventricle or atrial additionally what I've done is I've taken ventricular out of the stem and I've just put in cardiac arrest which is just as useful information in the stem there's times where you need to may need to do instead of two matched pairs three matched pairs in this case ventricular atrial and then cardiomyopathy are all matched and again this is a very balanced question these all diagnosis as well so very homogeneous answers and if you had the scan there it passes the cover test because by covering up all the answers you should still be able to interpret the image and come up with a correct response don't teach in the stem this is we're assessing not teaching so invasive lobular cancer can present as architectural Distortion right this is teaching it shouldn't be there so this was a pretty brief go through the salient points in item writing but these are well worth taking into consideration when you write your items they're going to perform psychometrically much better what I mean by performs I can make sure much better it means that they are valid Assessors of your exam takers abilities there are a couple of good resources for this the ABR you can contact they have a very nice manual on item writing you can also find this at the aur website if you go to the ASA section of that website you'll find it under the item writing resources there the nbme item writing guide which is longer but you only actually have to read like the first third of it and you'll get most of what I'm talking about as well as some of the cognitive background on why these things should be done like this is a nice guide that can be freely downloadable from the MBA website if you're just google nbme item writing guide you'll find it helps by