Signed Electronically Course Evaluation Made Easy

Signed electronically course evaluation

[Music] so Steve asked me a few months ago to give this presentation to talk about maybe some rationale of moving from paper course evaluations to online course evaluations because someone recommended that maybe that should be considered to the faculty councils and that was me I did that I was I was curious to see why we were still using paper evaluations and maybe if moving from paper to online course evals would be a an efficient maybe good idea for our evaluations in for the University at large so I'm going to talk about that today just sort of the rationale behind this and turn it over to Jason afterwards and then open it up for Question and Answer for 15 or 20 minutes so that we can kind of take note of what your ideas are if we don't already address those in today's session before I do that though let me say a couple of things first of all I'm not gonna be able to solve all of the issues that you're thinking of when it comes to online new vows there's there's no perfect evaluation system not paper not online we're not going to get that done the other thing that's not going to happen is I'm not going to convince every single one of you and that's absolutely okay too well what I'm going to try to do is address probably the biggest concerns that you're gonna have from me from paper to an online format and I'm going to start by answering this question right here why are we even thinking about this like what's with the proposed switch from paper to online and I think I can summarize the answer that question my opinion is that the cost of keeping the paper evaluations might actually outweigh the benefits of keeping paper evaluations all right and now let me kind of break that down and show you what I mean by this okay first of all I will address maybe the possible disadvantages of what we have in place right now keeping the paper evaluations and this is where my mind went when I first proposed in the faculty council if you know what what's disadvantageous about the system that we have in place the first thing that came to my mind was time right the first thing I thought was there's a lot of time involved in using the paper evaluations that we have primarily there's a lot of time used with scheduling you know with our administrative assistants they're having to schedule these classrooms in the days on our calendar they're scheduling student workers to come in or other faculty there's a lot of time being used just figuring all that out but that's okay but there is a lot of time in class that's being used to you know I know in my Monday Wednesday Friday classes I'm using about 10 to 15 minutes my Tuesday Thursday classes sometimes 15 to 20 minutes but it's a larger 60 Merson class just one day one sitting like that time I need that time but that might not be enough for you guys that I understand that that's not everything there's actually not enough time for students for by an ample comments either right about 10 to 15 minutes maybe students have a lot to say on those emails it don't feel like they have enough time to spell out what it is they want to say about our courses and some students feel rushed they feel like you know I don't have enough time to write anybody want to say or maybe they can type faster they can write and so forcing them to do this so so quickly in a classroom doesn't give them the chance to say what they want to say and I know in my evals I get a lot of na-na not applicable not applicable down the row they didn't have time or they were the last ones writing the evals and they just felt like they had to rush through it so they don't write anything at all I want their comments but the many most egregious of all in my opinion is how much time it takes to type all these things in by hand with our administrative assistants they spend hours and hours and hours and hours typing all these things in by hand when they could be doing a million other important things and I just want to hypothesize maybe how long it takes them to do this and so I wouldn't run the university net survey to all of the administrative assistants and ask them how long did it take you to type these in and collectively they spend last semesters 765 hours I'm absolutely lying about that number that's not a real number and making that number up I don't know what the number is I know it's a big number though okay and I know that they're spending a lot of time typing things in for example this came straight up with one of my evaluations right here drawing unicorn and a tree it was actually a drawing of a unicorn and a drawing of a tree but our administrative assistants can't draw that for us when they're typing it so they have to draw these very useful and spend their well-earned time typing this out which is honestly not productive okay so I find that the use of time for these paper evaluations is somewhat for lack of a better word wasteful I think we actually use a lot of resources with these - a lot of paper a lot of ink a lot of mechanics and copier that seems to break quite often I just I don't know our resources could also be better used when we have online formats to do this to begin with then finally this is one that I hadn't thought of before it's like talk to a few colleagues here my wife also is that with paper evaluations at least the way we have them now students aren't getting a chance to evaluate the entire course right we're giving evaluations in a fall semester between October and November we haven't ducted their final projects they haven't done the last unit of our classes they're not actually evaluating us in our teaching and tiring we're having to evaluate so early maybe because we need that extra time for the administrative assistants to type in all of those comments and it's a lot of comments because we in psychology have almost 300 students and and multiply that by our classes that's a long time comments that don't necessarily encompass all of our course time anyway which is kind of frustrating and I know that I want students to tell me what did you think about that final project what did you what did you think about those last units of material that we presented or discussed we don't get that on our paper evaluations at least the way we have now now I know you can do informal assessments you can ask in other ways but those don't necessarily show up for tenure or promotion we want those I would want those data in my full package being presented to the committee I want the whole class being represented you can't necessarily do that to wait another look but I'm not anti paper evaluation entirely there are some advantages to keeping it the way that we have them for example number one why we want to keep the paper evaluations is that you're gonna get higher response rates this way right if you're gonna get more students sitting in classroom taking the evaluation when you know they're in the room with you as opposed to hoping they're going to do if you send it to them online or in an email right maybe maybe that's probably one of the most the biggest reasons why some of you are are opposed to using online evaluations because you're afraid and maybe maybe my students are not just gonna take it and a lot of you know people from other universities have said that I sent across someone in a conference just three days ago in Atlanta and she said we use online evaluations and my response rate is 25% it's awful she said I don't know what to do about that I'm sure you've heard that too from other people she said what do you do and I've got an answer for that but we'll get to that here in just a few minutes twenty-five percent response rate you can't do anything with that you can't delete even from students what they really feel about you or the course there's no meaningful generalization you can make from a small sample size that's not gonna work the other issue is that maybe we're worried about having trustworthy accurate responses you know maybe if students are getting their evaluations through email or doing it through some online system maybe the data that they're giving us are not accurate maybe they just go through and they christmas-tree their evaluations they have their suitemate come along and they do it for them right maybe we just don't know if what they're giving us is usable actual real good data maybe what I want to do in the rest of my time up here is to talk about those two advantages of paper evaluations and discuss why online evaluations can maybe even do better than that why these are not actual concerns from moving from one system to the other and I'm going to do that based on number one past research from other institutions and number two based on my own experience and Jason's experience as well so let's start with the top question okay maybe maybe you're wondering this right now how well online evaluations change my response rates and the quality of those responses because I'm worried if I leave it up to them to do it on their own maybe I'm not going to get a lot of responses and that's what I need for T and P right we could get that so here's the truth this is honest okay the research has been remixed on this response rates might change if we move from an online paper to an online the research is not very clear some research is showing response rates as high as 89% others as low as 31% it depends on what you're doing it kind of depends on how you're trying to encourage your students to do this but that's a really large gap high and low response rates and I know that I can use a thirty one or a fifty or even a sixty percent response rate I need as much data as I can get this is a little concerning to me and what's even more concerning is that the quality of the response is also mixed according to this research it just depends some research has found that you get better results by using online methods that students provide more positive feedback more helpful feedback and comments other research says no not so much students are likely to be more negative or they're gonna provide less helpful comments we're just gonna go through quickly and provide no comments at all and other research says there's just no change it doesn't matter what you do paper online students are going to be the same research to be a little bit disconcerting until I think about what these different institutions what these different researchers are doing and what they're looking at and you have to realize that each of these institutions are using different methods of encouraging their students to do these course evaluations that's something I want to talk about near the end of my time up here is how do you get from 31% 89% where my colleague did in Atlanta three days ago how do you go from 25 percent response rate to what I had when I was at Alabama and 95 percent response rate with online evaluations how do you get there and I think the answer is right here it depends on how you tell your students to complete those evaluations let me elaborate a little bit further on this because I know this is a big big concern of yours in this room okay the study was done in 2010 analyzed over 4,000 online evaluations and they wanted to know is there a significant difference between the response rates and quality from online to paper evaluations over a three-year period and here's what they found not rising number one when these universities were involved with a study made the switch response rates dropped at first it kind of makes sense to me right I mean if students are really used to coming to a classroom and completing paper evals in person when you ask them to do it online but maybe the university doesn't have all of their procedural ducks in a row and these students aren't used to it and they don't actually know how to do this or how to do it right or to get all the responses but after a while that became less and less of a concern over the 3-year period when the researchers eventually found that there were no significant changes and the ratings of the courses or of the instructors that's good news no difference between one or the other but honestly most interesting to me is this next funding for this this three-year period is that online evaluations also produce more comments and lengthier narrative comments then did paper evaluations and this is not just the only study that's found on this I have citation after citation of the citation that'll show you the same thing when students are given a chance to give you their feedback when they can type it they're going to give you more feedback than if they're gonna write it and I know why that is because maybe I'm part of that millennial generation and I am I would another type something gonna write it by hand I can type a lot faster that I can write with an ear carpal tunnel hand you know I can't and so I would rather give you feedback as a student or other than my students give me more feedback this way and students this is a technology generation these students wanna type more than they want to write and I think that's why they're gonna provide more data for us through these methods now other universities have done something similar they've moved online evaluations over the last 18 meters Northwestern being one of them and I know before you're thinking it we're not Northwestern we're not going to compare ourselves to them this is just sort of a case study way we think when we do think is very very different than them but just as an illustration Northwestern moved to online evals and nine and what they found was that the number of comments that they had increased how many comments the professor's got and I'm useful those comments were for their teaching after just four years implementing these online evaluations and that's not just Northwestern that's many many other institutions across the board a lot of the universities are supporting this idea if you Institute it right if you have the right procedures in place you're going to get better data using online evaluations so you know can I actually believe what these results are giving me why do I know that I'm gonna be able to use these data or could it be someone sweet made a roommate that's a great question my answer to all of that is yes I do believe online evaluation results can be trustworthy if we do it right for example study in 2006 found that students who used an underlying network but you can notice that voluntary voluntary online evaluations were marginally more likely to choose both higher and lower ratings on the likert-type scales they had on their evaluations it's the same kind of finding that we know when we look at sites like rate my professor the students who use rate my professor there is a certain kind of student they're not your typical student they're the ones who are really really any rate at you because you gave them a bad brain they didn't allow you and they're gonna tell the world about it they really really loved you and they want to sing your praises to the Internet right it's not the student in the middle of the road it's not the student who is ambivalent about the class they're more likely to choose the highest and the lowest ratings it's the same thing that's happening here if you give students a chance if you stay what if you want to it's up to you if you have time please do this for me if it's voluntary the data are not necessarily positive and those are not necessarily the kind of ratings I always want on my evaluations I want all of the students data not just maybe the happiest and angriest but even more so a study in 2010 analyzed a lot of data over 600,000 classes over a six-year period using paper evaluations and 53-thousand classes using online evaluations to also compare the trustworthiness of their results and they found no significant differences between the two again depending on what it is that you decide to do how they decided the trustworthiness the accuracy absolutely no difference so no one will be worried about their online data being misrepresented or somehow compromised in these studies it shows it's all the same I want to tell you about my experience with online evaluations in the detailed version of how we get to an evaluation summary for each of your classes so it's kind of hard to see but is just a ton of steps and it gives you a better perspective of maybe all the work that's going on behind the scenes that may be that you that you've not seen before so the first thing we do each year is each semester is an institutional research we have a a system that's been built that pulls a list of all available course sections for the given semester they could be evaluated and identifies whether the little checkmarks that goes alongside the course section is is left unchecked because you guys get to select which course you'd like to have evaluated if they're non-tenured it just checked them all and obviously there are situations where that could be the chair and so the the general data is presented to the departmental secretaries you through Puerto and they're able to work with you guys and with the chairs to select the official forces that want they want to be evaluated that data is actually connected to the system we have so once the selection period is over then we're able to take that data in our course evaluations are printed by Stacy and her and it's about forty two thousand sheets per term that we're printing and so not only do they have to be printed but they have to be you can imagine for really one person to do and so there's a certain period of time where that work takes place and then it's done that gets delivered to the departmental assistants and then they go about the process of scheduling the evaluations and you guys know what takes place there and so the valuations are conducted and as they are being returned to the departmental assistants they are separating out the written comment piece from the quantitative piece the quantitative pieces but back in that envelope and delivered back to institutional research hopefully on a similar basis not all at once at the very end because somebody then evening those day and going to one of our copiers and scanning them and so basically we have what's called an om our solution optical mark recognition software that takes a blank version of that survey and compares it to the filled out version and we through the software can train the analysis tool to say this is question one and here are the options available and this option is strongly agree and it's worth five points and this is and so basically it compares the blank copy to the fill out copy and decides whether or not that answer was a five of four three two or one if it can't decide because somebody marked two answers and literally if anybody ever marked two answers and crossed one of them out and left the other one then it will pick it up as an error and basically Stacy and Laci can say oh there's an error there so in and what looks like an Excel spreadsheet they double click it and it pulls up the image and they can decide that was supposed to be a three or that was supposed to be a five and so through the course of perceiving and scanning these these evaluations they are cleaning them one by one I don't know what percent I'm gonna give in class there's probably less than a dozen responses that need to be clean but you can imagine we're getting about thirty five thousand responses back every semester so it's a lot of time just spent in there cleaning the data that's important and so once that data has all been received from you guys and it's all been clean then we have a code that on the back end that goes and runs that data through some checks the seed is the courses that have been returned that we know what that course is because we need to know information about you are you can hear what department you belong to and that's not encoded on the course evaluation itself and so it's basically pairing up the data to what we expected to receive back and if there's any issues then we have to work through the process to clean that up and then one that's clean then we can run our analysis and so about three or four days before you receive them we run the analysis into our data tables in what week our staging environment and we look at that so we go through a process and making sure that looks okay and assuming that it does minutes sitting there in staging until we're ready to deliver them to you and basically we have a little code that we run that just pushes it into the measurement that you can see import ago we do that at 12 o'clock 12 o'clock on the day that you're fine the final great review and so sometimes that's the we never see so here's something to consider I have data dating back to like 2001 in my database on course evaluations you may or may not be able to see that certainly if you haven't been there here this long that you can't see your results from this but I have Dayton going back that far but I have zero qualitative data so all the written comments are only stored by the department and so that data is completely disaggregated the qualitative data does go into the assistance you guys know that process they hand type them up as describe it then reduce those reports to you separately I guess to email and again that's the only place that those exist and so that's what that's the process so how does that look for us so I'm giving you two here you've got the fall semester of training over the last four years in the spring semester training over the last four years as well the blue bars represent their response rates that we're receiving from your class and/or from from the total undergraduate population we have different kinds of evaluations so this is including what we call red one that's revision one that's the official survey that he's given to most classes have a music service survey which is different and then we have the our you know G for the religion class and so there's three types of evaluations that are distributed on campus and so these numbers include so you can see we average about eighty three percent response rate the day that the evaluation first of all that only includes the evaluations for the total potential evaluations of those classes that were selected as being administered not this cross listing and so the numbers actually inflated here so if you have a class that's cross listed we don't want to give you two different evaluations because we don't know who's filling them out you know the students when they sit there in the class I don't know if they're so sure site right so we choose the primary one and we print enough to cover for both but when they come back in I'm only giving you one response back and so it's very difficult for us to say in total there were a potential number of this many people in the classroom because of these cross listings and as a result the number that I'm using to say the possible number taking the evaluation does not include cross listing courses if we were to take the total population of courses that that you said wanted to be evaluated and the total number of responses we got back the actual response rate is more like 65% and that's not accurate either because sometimes you tell us that of course is going to be evaluating that never gets evaluated so the response rate is closer to 83% but not quite it's probably more like 78 to 80 percent so we can't really get a given firm number the spring those numbers look about the same although there's been a little bit more variance over the last couple of years we do something different in the DAL and this is my last piece here but obviously they don't come to campus there is no opportunity for face-to-face evaluation we had this optional for the first few years of the DAO and the response rates were terrible and it was this kind of thing hey go take your evaluation and we appreciate the feedback and we were getting I don't know 30 percent response rate maybe maybe 30 percent response rate so we really thought about it then we created a new system we've built our own so we didn't buy something we built our own that's a hybrid of what we're doing on the traditional campus but we're getting basically a hundred percent response rates from students finishing the glass okay so because it's taken at the end of class and if you're not in the class and obviously you're not gonna take this but here's what we do on a seven week model and so each course is evaluated and we add essentially what Moodle calls a unit between you between six and weave seven so in between those two is this unit which is the course evaluation okay and so then what we do is we say a hi week seven until the course evaluation unit is done so they can't see the final unit of their class until they go in and taking the course evaluation and so why I say it's nearly a hundred percent completion in a theory you can't finish the class until you take the course evaluation but here's something that we don't publicize but it's true and some people I guess either know this or they accidentally know it if you click on the unit link what we allow is you to then immediately be able to see week seven so you don't technically have to fill out the evaluation in order to see week seven you just have to click the link which opens the evaluation so potentially you can open it and they shut it down and they have access to week seven so what we know about this now is when the students click that the URL we embed into it the code that's necessary for us to know who it is that filled out the survey we can choose to include that or not but we would know that this person didn't click this and we could pass that into the survey itself and the hidden field so that when they hit submit it also marks hey this person actually filled it out and completed it and so now this model works really well for the DAL it wouldn't necessarily work well for the traditional campus and its exact form but a version of this could work according to whatever you know the institution wanted to do and so essentially what we could what we do is as soon as they they click that and move on to it then week seven available and then the other advantage trust then is that data that goes into the system that we are scanning the other data into and so we skip the whole process of cleaning because the data is coming in clean and so that saves a ton of time as well on our staff side because they're not having to clean it the third piece that I'll mention and I think I've done is that all written comments also come over and so we've already built the process because we have 800 students every session so twice a semester they did is twice this semester we have 800 students doing this already and the written comments are coming over as well and then we've written the code to push that into a process where the department's themselves in this case just the dao can actually execute a report and see the written comments alongside their quantitative comments we have experience with this and it seemed to be working really well he talks about us sort of a start-up cocoon well we've been doing this for three years or so so there there's a model out there there's just obviously some things that would need to be worked out in terms of making it unavailable to the on campus paper and so I was given the option though how I wanted to encourage students to evaluations find three ways that I could do was one of which we've covered so far and that is voluntary we should ask my students you know the goodness of your heart if you have free time would you just do me a favor fill out my evaluations for me that would be great I didn't do that I wouldn't do that I wouldn't trust my students to do that okay I had another option incentivize my students to complete the evaluations and a lot of other universities a lot of models out there use incentives that's what we do now for our advising evaluations and I don't know if you know but our response rates have skyrocketed since we started using the incentive of earlier registration right so that's one method that you could consider to encourage students to do this and that's not the method that I chose either I showed the third method some research says for me it worked and that was I made them I made them I made it part of their course grade I had the option and I said it's one percent it's one percent of your final grade in this class if you complete this online evaluation and all you need to do is give me a confirmation sheet that says you know you completed it this is your whatever your name your number whatever it was and give me that sheet and I'll just give you that ring and that's how I knew that they actually completed it and my response rates were 95 to 99 percent every single time I did this in online format at the University of Alabama now this was this is not just my experience my wife have also taught at Alabama she had it to where she told the class if 95 percent of you complete your online evaluations you'll get extra credit so that there's buy-in from the entire group if all of you do this and we can show this percentage then you're all going to benefit from it and she had nothing less than 95 percent response rates on her evaluations - and I know that in my experience my comments really were lengthier richer than what I'm getting now I just look at my evaluations last week and there's so many not applicable for blank spaces or completely useless bits of feedback bomb.com I get that on every I don't know what that means I don't understand it but I wanted to go away and I want real feedback on my evaluations and I saw that when I did at Alabama and I hope to see that here so in short here here's kind of the argument that we're trying to make here in short I think that online evaluations can actually save or produce a lot relative to keeping paper evaluation the first thing that comes to my mind again is time and resources I think we can save a lot of time with data cleaning and process and sorting and scheduling and typing and everything else thing if we just move to something that we already have in place online evaluations give us also a better picture of what we're actually doing in the class because right now paper evaluations don't do that and I won full a full picture of what I'm doing in the class and I want my students to give me good data that I can use to be a better teacher and that's the thing about all this by the way I'm gonna be transparent I don't I'm not worried about tender and a commotion in terms of all this stuff that's not my that's not my concern for any of us my concern is that we get a lot better more data and makes us better teachers you know that then helps us get tenure and promotion and all that whatever but I really want you more data so that I can hear my students say you know that end of the course really didn't fit it didn't quite meet the objectives that you had and those assignments didn't quite fit but I don't forget those data right now because our evaluations don't fit the whole course we also find that online evals give a similarly high and sometimes higher response rates depending on what it is that you choose to implement we get trustworthy accurate results there are mechanisms that are in place that we used at Alabama to ensure that it was in fact the actual student in our class who was completing the evaluation and not someone else but again that depends on what system we put in place and maybe my favorite is that we just get better day more rich narrative comments more students that we can use to be better teachers and make our classes stronger so that's kind of our case for this consideration McGinnis team said yesterday we're not necessarily making a decision right now we just want to hear what you have to say in response to some of these ideas which leaves us for the next maybe 20-ish minutes to open the floor for your questions and your comments and take notes we're gonna try to address the things you have to stay but we want to know from you what do you think good or bad thumbs-up thumbs-down late on so I said I was gonna come in and be the boisterous minority but up until seven minutes ago you had me sold I mean I was convinced cuz I hate how much paper we use I hate the time I hate the resources but I've got a couple problems one is just one is I'm not going to give my students extra credit and I'm not gonna take points away from doing this and if it's voluntary they're not gonna do it right so I don't have a solution there the DL system works really well if there were a university system that solved that problem I might be willing to be convinced but I'm not going to give them 1% or 0.1% because I'm just thinking of that student freshman comp whose point four away and they passed that class because they filled out a course evaluation including the final exam which let's just say takes place Tuesday afternoon from 3:15 to 5:15 and I have to turn in Senior Grade the next day how exactly would I give them points anyway by the time I'm grading everything else and there's the third thing which is I'm still not convinced they're trustworthy that they're not just going to be sitting in their dorm room and saying hey you've had brown before what did you think about them and nothing in here shows me that they're proven you know maybe the students know that within mid-november they're gonna have evaluations I know that it's coming so what do you think about Kevin Brown you know I don't really know what to write on here so they're friends opinions can still inform them even when they do not hate but in these cases in past research they're giving us more feedback than less so I wonder if you know that same concern to be made with paper evaluations remind me what your second question we didn't we didn't put the evaluation after the final exam we let it go I think they were able to take it within the last two weeks of class anywhere in that period but it usually always fell before their final exam so the final exam would not be included on their evaluation of the course which is a problem but it's also better than the last month month and a half them we're getting now although personally I don't mind if my final exams not being assessed I mean it's a final exam there but the rest of it one would be attaching to this and going and putting in mailboxes that the students don't use and so then the post office is spending enormous time at the end of the semester looking up people's pass codes to their no bikes they don't use so they can get the one piece of mail they care about for the whole semester and so it's conceivable that there could be a two checks this dump there did you fill out the evaluation and did you is your bill at the level that it needs to be and through portico they could be delivered then some sort of bins additional paths that you could also see as a faculty member if you want it through Porter go to check or I don't know something like that could potentially work just rush through it so that they can get something and do you want to give it negative sort of consequence to something so important so you didn't fill out your evaluation we need to do it before you take the final they go through a quick they're angry and what happens right and so obviously these are concerns that we'd have to talk about but there are possibilities and it's just one let me say on the DL tanzir the other questions on the DL side of things a little different because they were already filling out online evaluations as a voluntary thing and then went to sort of a required and we did worry that they would get to this in and be frustrated that they can take them and whatever but we saw basically no change in the actual quality of the evaluations with one exception and that is is true the DA else does their writing really three weeks left research what what is the central what is the the import of the evaluation for for tenure for promotion senior faculty review and that type of thing in other words people that have gone to the online to the action are they at the center of what gets a person tenure or not and do we have to do quantitative or qualitative evaluations together do they have to be joined well I guess the first question on that is the teaching emails are not as central in these studies as they are to us and stay sort of comprise the primary method getting some promotion but they are an important component a lot of these larger universities you can find that publications or conference presentations would also be in conjunction with teaching evaluations central yes but as central as us you know because there really are no compensatory right factors or elements for teaching evaluations now that's true and of your second question do we have to be quantitative and qualitative I don't know if I can answer that question in other words could you could you do it's a speedier way to do the quantitative and then do online the qualitative which would which would get to the narrative and the longer narrative yes I guess that's probably a big question I think you've separated the two out it does I mean keep in mind for you I'm asking that Jason's because you know we do a lot more reflective valuations as we go through the semester yeah so that qualitative evaluation where we ask them to write a narrative seems more in keeping with what we've been asking them to do well at least multiple times during a semester and if that couldn't be done in class the last the last day or something and include some of the final elements of a class separate from or maybe doing separate from doing the the quantitative one you know yeah those comments are read in their entirety and I've been questioned this comment second year this semester they become more extreme the quality is the same but do you have more extremes because really the evaluations where aren't as high stakes and so - but they also used I'm concerned you know by giving the opportunity to wait just that my last exam I didn't like that last exam now as opposed to I'm just kind of concerned about being able to defend evaluation given that the on paper evaluations right now are also anonymous and students theoretically have lengthy amount of time to provide whatever comments they want and there are extremes when including the unicorn tree example we have a lot of those in there how why would an online format produce more extreme responses in our students I just see that I just see our students is that different though I wonder because if we're telling them that and this is one thing together the researchers said that I didn't mention is that if you don't want to use the voluntary or the incentives or the mandates to complete the evaluations one thing that research says works it is just tell them why you're doing it right you just explain to the students that you know these evaluations are not only important for my promotion but it's also important because you know I need your feedback to make a better course for better judgment you know one of the things what I like about doing it in classes because I have my faculty doing that and I we take that time to give that instruction research the issue of voluntary voluntary surveys there is a randomness to it accounts for that that's a good question Nancy my knowledge I don't know how the literature addresses that I know that the literature definitely frowns on the use of purely voluntary methods for these reasons in her low response rates and you can excuse that's true that's true but skewed but also still I mean we still get somewhat skewed data with the students who do decide to show up on the day they do show up there's some kind of selection bias at the student sewer I understand the randomness that's true because we choose the day and they don't know when they may not show up that David it's also skewed because there are certain kinds of students who are showing up on those kinds of days I don't know I don't know no I didn't see anything most of the research has been published the sort of stops around 2010 to 2012 and I wonder if that's not because I think I heard the statistic in the reading or maybe talking with Jason that about 90% of colleges and universities of adopted online evaluation methods now and I wonder if because it's sort of evaluating a handout needed and I can get that all done in a class in the same time period thank you text fashion to make them right and I would get increased I would feel like be able to present the importance of that I have a response rate those issues I mean my greatest concern seems like volunteers a bad idea that's basically