Sign Ohio Assignment of Partnership Interest Easy

Document type sign assignment of partnership interest ohio easy

work that they've been doing with one of their districts uh that used a remote learning platform this spring and they were able to analyze some of that data and um understand a little bit more about how data from that platform can help us think about engaging students and thinking about student participation so our hope is that the hearing from them will generate some ideas on our end and help us think about the kinds of data that we all have access to and and how we can use that to support district schools and students um so in a moment i'm going to turn things over to dave hirsch and lisa son bon matsu who will present this morning just a few sort of technical issue technical notes um i want you to all know that this webinar is going to be recorded and we will share that recording after um the the presentation so that if you want to watch again you'll be able to and if some of your colleagues are not able to catch it this morning they'll have an opportunity to see it as well we'll also be sharing the slides um after the meeting so you can kind of take a look at those during the meeting itself during the webinar itself please feel free to um engage in conversation in the chat box if you have questions that come up we'll certainly be pausing um throughout to address questions and try to set aside time at the end for questions but throughout please feel free to use that chat box and we'll keep an eye on that um and address questions as they come up we really are looking forward to an engaging discussion with all of you about this data and its implications and uses um the you uh should all be i think i think you're all muted at the moment um we just asked that folks remain on you while you're not speaking um and uh you can see sort of in the presentation here there's a few tips on how we can help i'll all help make this presentation go smoothly um go ahead and turn off your videos that'll help with bandwidth um and then you can follow along and tweet about today using these twitter handles here on the screen um again i don't want to say a whole lot because i want to preserve our time for dave and lisa to share what they have been working on what they've learned and have some discussion with you all but just thank you again for spending your time with us this morning dave i'm gonna go ahead and turn it over to you awesome thanks so much heather uh and uh i'll just echo your thanks to everybody we know how crazy a time this is for everybody involved in education and uh we're just happy to see so many people on um uh and as heather noted uh i'm joined here today by uh lisa san benoti who is our analytic lead at the center for education policy research i'll be doing most of the presenting but uh lisa will be sort of our point person for any technical questions that arise she and her team do do all the kind of deep dive analytic work um we also have a couple other folks from uh from the proving ground team here uh on and uh some of you may have interacted with him already or seen us at various events um but uh we have uh jen ash uh and katie kinniger uh on as well uh unfortunately in my presentation mode i actually can't see the screen at all so i have no idea if they're waving at you or saying hello um uh but they are also on and available uh jen and katie work with our 19 rural partners in ohio uh and then we also uh i hope uh but i can't see have uh amber home pat note on who works with our three uh larger urban partners in ohio um are the i should make a quick note uh because i can't tell if she's on or not um but the analyst on our team who did all the amazing work that i'm gonna share with you today uh and makes me look good by giving me something to share with you uh her name is rachel lee uh rachel if you're on um do you feel free to turn off your uh turn on your video and just give a quick wave uh but rachel got sick this morning so she may not actually be on i don't know yet um but rachel did all the all the hard work here um as heather noted um we will be relying mostly on the chat window because there are so many of you uh and also because as i noted i can't see uh see if anybody's talking so if you're waving or hollering um i won't know it so i'm gonna pause periodically and the team's gonna monitor the chat window and just make sure we don't miss any of your questions uh don't be shy about asking questions throughout while we do have uh plenty of content to cover i think the most important thing is being able to to have that interactive conversation and answer any questions that you all have uh the content's not going anywhere and so if we don't get to everything um uh we can always follow up more later and we can uh as heather noted we'll be sharing both the video and uh and and the deck and so there's nothing on here that you won't be able to see if we don't get to it in the uh 51 minutes we have remaining uh so with that i'm just gonna dive uh give a quick overview of what to expect uh of the presentation um it's a little ambitious for 51 minutes but uh but i think it's i think it's manageable and i think we tried to hit the sort of key points uh or the minimum needed to to kind of take next steps uh i'll note that the goal from today's meeting is not that you'll come out of this meeting suddenly uh suddenly able to do all of the stuff that we were able to do with this partner we've been working on this for a very very long time the goal rather as uh as heather highlighted is uh to offer an example of something that is imminently possible with the data districts have been generating and are going to continue to generate over the next year um and uh and sort of plant the seeds uh for you all of what's possible so that if you're interested in doing it uh you can you can follow up and kind of take next steps and so we're uh we're gonna dive in in a few different ways but we are gonna cover not just the outcome of what's happening but also the sort of how the how we did it and some considerations for doing it yourself um so the big picture overview is uh one of the many many unfortunate consequences of profit 19 is that it left educators without many many of the data points that uh we have come to rely on as educators and maybe we don't even realize just how much we like we rely on them and uh while there are many many data points like that one of the most critical that we've lost is actually maybe the simplest data point which is attendance you know our students coming to school uh we use that in so so many ways we use it as an early warning indicator to identify students in need of additional intervention we use it as a predictor of test outcomes long before you've administered a test obviously there's all kinds of truancy implications and your house bill implications of attendance um but from our perspective the most important ones are sort of the formative ones right what do we use this data for uh to inform uh inform the action we take in support of students so insofar as students stopped physically coming to school most states like ohio decided to waive the requirement of reporting attendance in the standard way right districts were essentially marking every student as present for regulatory purposes for the rest of the school year and that left us with essentially none of this data none of this amazingly useful uh formative data uh that we've had for so long um and while we didn't obviously have a ton of time to to rectify that or kind of fill that void uh perfectly in the in the few months after uh after school shut down for covet kovat19 um we were able to do enough to start building a picture of what is possible for next year and how even if we end up in a hybrid or even fully virtual world for some of some of next year we don't need to be left in this vacuum in this void of this this highly usable data so i'm going to share just one example of the type of analysis you'll you can do with school district data and i'll note that these are data that is possible with uh with what districts are collecting sort of anyway um and and i mean that in in two ways one is based on the recent um kind of very thoughtful very strong guidance ode recently issued on tracking attendance for next year your your school district should be should be capturing this data in their sis somehow but also because they've started to ramp up for at least the possibility of some part-time remote learning uh districts are going to be using platforms that are automatically generating the type of data that we're going to share today and so in either case you should have the data exists and so really what we're talking about is uh is how to how to engage with that data in a way to make it actionable to districts in a way that's sort of analogous to how attendance data has always been actionable um so very very brief agenda i've sort of uh mentioned this already we're using one example of what is really an infinite number of possible things you could do with the data that districts are going to be generating over the next year or so um but i'll give i'll dive into the example so you can see what's possible what we did with it i'll dive into the data sources we used to create it the methods and tools we use to to create it uh and then we'll close out with some basic uh considerations as you think about things you might do with data you're getting from districts or things you might do to help districts with their own data um these considerations are useful as you as you consider those things i'm gonna pause there just uh since i can't see the chat window i'll just pause and let somebody holler at me if there are any any questions in the chat window at the moment there are no questions yet awesome thanks uh so i'm going to start as noted with the results of the analysis that we generated um and uh i'm going to go through sort of a quick bullet point overview and then show uh show some kind of pretty charts that uh rachel lee generated for us on our team um and again i'll just note uh that the reason these charts are sort of think of these as inspiration for what's possible not necessarily recommendations for the the exact or the only thing you could do uh so the overview is uh we we kind of broke the analysis up into two pieces the first was uh the partner we did this with was interested in knowing sort of who was logging in when were they logging in and are there patterns regionally to where they're not logging in and the regional pattern piece was important and sort of distinct from how attendance has usually worked in that there's there were obviously many hypotheses about students not having access to internet in certain regions of the city things like that and so being able to impose a geographic lens on this was actually really important so i'll be showing you sort of what we did and we'll talk a little bit about how we did that um but in addition to wanting to see where and when students are logging in we wanted to start to close with them this knowledge gap around what it means if students are not logging in so we have all this knowledge this kind of very detailed historical analysis based data on uh on what it means when students don't show up in school right what is that absence of physical attendance uh uh mean for students and the short answer to that one has always been it means a lot right we know that when students come to school less their test performance is dramatically dramatically affected their sort of social and emotional um well-being is often affected at the very least attendance is a predictor of even if it's not a cause and sometimes it is but even if it's not a cause it is very often a predictor of uh of the types of things uh that we care about as educators um whether they be academic or social emotional um we don't know if that's true for logins right we don't know if that's true for engaging with remote learning we don't have the historical data on it and so we have no way of saying if a student misses ten percent of virtual days does that mean the same thing as missing ten percent of physical dates um and so the second analysis we did was looking at the patterns of login behavior and the relationship between those and prior absence patterns um and our findings there suggest that not terribly surprisingly um they're so related that there's reason to feel confident that students not engaging in virtual learning is providing exactly the same warning signs as students not engaging in uh sort of in-person learning um and again this is very preliminary data but when you see it i'll kind of go through go through the inferences we drew from from that data about that um so without belaboring the bullet points that you've all probably had an opportunity to read while i've been babbling i'm going to dive into the the sort of deeper dive of the analysis so the first part is i noted where and when are students logging in uh and we started geographically so uh in some census tracks uh nearly a third of students never logged in in the four-week period that this analysis is for uh we've actually done this analysis now over a much longer period than that but we're showing you the example just a kind of discreet four-week chunk example um but a third of a student's never logged in at all in four weeks and where that takes place is super concentrated in regions of the city and one of the things that happens when you show this to the district and by the way i should also note all of this data has been modified the patterns are the patterns are correct the findings are are legitimate but we've modified all the data to hide the district that this was for uh including this geographic map this is not the actual map of any city in the united states just in case somebody's wondering about that we've modified the map to make the patterns remain but not not allow it to be identified um but what you what the districts see when you show them this is they know exactly what that really dark purple area is they know exactly where in the city and they immediately begin to generate hypotheses about why they have such high rates of not logging in at all um uh and you know some of them are obvious some of them we had to dive a little deeper into and i'll share that in a little bit um but for the most part uh districts weren't surprised at the fact or the pattern that they saw here that you know this this area that my laser pointers on here of the city was more um had higher rates of students not logging in in this area had much much lower rates um they were a little surprised by the degree of that right if you think about um even the like highest that your most absent students if you have a 33 chronic absent rate you know those a large portion of those students are still showing up nine out of every 10 days right they're missing 10 of school but they're missing one day every two weeks what we're seeing here is a third of students one third of every student in the district didn't log in a single time in a four week period right so we're talking we're seeing just a degree of difference that's that's much bigger than anybody sort of anticipated or at least anybody hoped um when we break that down by grade level um we sort of we sort of see uh similar patterns to what we had seen in uh with attendance but um but again the extremes are a little bit different so uh one thing to note is that the you know in pre-k um attendance rates are generally a little bit worse uh or sometimes much worse than even kindergarten um but here we see a complete lack of engagement with with online there's so many factors uh that go into that not least that pre-k students aren't predominantly using virtual or online tools to do remote learning uh and so this is not by any means uh evidence that like they had the opportunity they had the access and parents didn't um so really what you're seeing here is also evidence of how the district rolled out virtual learning um but the point is not necessarily for us to draw conclusions right the point is for us to present data that the district can use with its own knowledge to generate uh generate its own conclusions about what they're seeing or at least generate its own hypotheses uh to dive into a little bit deeper um we were curious if the two patterns were sort of consistent which is to say the pattern the regional patterns and the grade level patterns uh were consistent and generally speaking they are uh what this data this deck is showing or this this slide is showing is that the areas of the city that have the higher absence rates or the higher non-login rates are basically the same but the degree of not logging in is much much higher in k to two um and so that's not exactly surprising given the first two slots um but this is when we started to get into the when question uh that is really important for the district we work with in developing its plans for next year right they're planning at least some portion of virtual learning and they want to know when they should focus on uh assignment delivery when you should have zoom meeting zoom classes that kind of stuff uh and so we were able to unpack this a little bit by looking at uh not only what day of the week which is what you see here the login rates were the highest uh but also the um the times within a given day that login rates were the highest um one thing to note about this is this is not reflecting a lot of synchronous learning right there's not a ton of like concurrent zoom calls taking place because this reflected the first four weeks of online learning and in those first four weeks the district hadn't yet set up a lot of synchronous uh learning opportunities so when you see login rates in the vicinity of 33 to 34 on any given day that doesn't mean teachers were having classes with 34 of students showing up um it does however mean that on any given day 60 something percent of students were not doing any online instruction they weren't logging into anything at all um which is interesting the other interesting thing i think here um is that we have a huge drop-off in engagement by friday uh and so what you see um uh on especially by that fourth week is uh rates in the like seven percentage point lower for friday than on monday of that same week uh and that's a pattern that we've seen kind of be consistent across but it does seem to be getting a little bit worse or seem to get a little bit worse towards the end of the year and that has obvious implications for uh for how we structure assignments and things like that uh going forward likewise uh we see that logins are peaking uh in the middle of the day and this again is not enormously surprising um but this is true across all grade levels and i'll show the next slide in a second because this pattern is basically the same no matter what grade level you're looking at um but what you're not seeing is kids logging in at the time that they would have shown up in school if a high schooler started school at 7 15 they were not logging in at 7 15. right your high schoolers were logging in much much later in the day with your sort of modal login window being between 11 and 11 and 2 pm or so and again this has implications for uh if you're going to have synchronous synchronous opportunities when should you have those synchronous opportunities probably not at 7am um based on this uh in fact you see way more students logging in at 11 p.m uh than you do uh logging in at even 8 a.m and so that again that has that has some implications uh before i get into the next sort of chunk of slides i want to pause and see if there's any any questions in the chat window at this moment hi dave uh there are no questions yet and i just want to encourage folks to use that chat box as you um have questions or reflections i know um for me personally this uh being a mother of two children um these slides resonated with me as i thought about when we logged in and so um we'd love to hear your thoughts about this what resonates um and your questions as well so feel free to use that oh we do have a question what is the sample size of this data uh that is a great question um the uh i do have that uh lisa i'm again i can't see the the window with everybody's faces but lisa if you're on and you can chase down the exact sample size um uh please please either just holler out or put it in the chat window um i can tell you that this is a district uh of uh about 28 000 students um and uh the so that's roughly the the sample size uh which is to say that's roughly the denominator in all of these uh so when you're looking at uh sort of shares of uh like let me go back here um the share of students logging in by uh by a given day so on like monday or wednesday april 8th if 32 were logging in that's 32 of all enrolled students so 32 of 28 000 or so uh roughly 10 000 students give or take 9 000 students in each one of those bars um but if we can we can probably get that exact answer so um if somebody on our team can chase that down and drop that in the chat window that'd be awesome we also have a question from sarah are the logins represented from one platform or multiple platforms perfect question yeah we're that's one of the considerations we'll go over in the end this is one platform uh the platform we're working off of here is called clever uh which has i think something like a 40 40 saturation rate in ohio something like that so not a not a majority of districts using it but clever is a single sign-on platform um that essentially captures districts used to route uh logins to many different platforms so if a district's using you know iready or map or any other sort of online learning tool students are logging in through clever um there's a little bit of a hole a little bit of a hole in the data for this district in that um just some teachers did use um google google classroom which is not captured by this and some teachers did have zoom meetings with students and neither of those are routed through the clever single sign-on platform uh so there's a little bit of a miss but in this particular district um that was a big minority uh a small minority of the students was um uh was user teachers were using that platform um but one of the biggest considerations for how to process and analyze this data is uh is how consistently the platforms that are generating the data are being used across a whole district this is infinitely easier and we'll talk about this a little bit later but this is infinitely easier if the district has a single platform that everybody is required to use across all grade levels and you can just pull the data in one shot um over the last three to four months that has not been the case by any means there's been sort of a uh a buckshot approach uh to you know use whatever you have get stuff out to students and families and do it however you can and maybe capture some information in like a google sheet or google form on the back end somehow um but going forward what we're really expecting to see just from our work with partners is uh the much more consistent use of uh much more consistent guidance across districts of tools and uh and i'll know a little bit later but the easiest thing to do is actually not necessarily a single sign-on platform like this but the learning using the data on the back end of a learning management system uh to capture logins you could structure the exact same analyses but a learning management system would likely allow you to capture a lot more of what's actually happening than a single sign-on in a place that has sort of multiple ways you can multiple ways you can log in we've had a couple more questions and thank you lisa for responding to some of those in the chat box that's really helpful um we also have a question uh from andrea did this district have some buildings that were doing pencil paperwork during remote learning did the district provide one-on-one devices during this time uh yes to both um and i i should qualify that uh that i don't have at my fingertips like flawless data from this district on all of the other things that they did um but i can answer generally that yes some uh some grade levels uh particularly the younger grade levels we're using um pencil and paper uh which is again reflected in sort of that pre-k in kindergarten like particularly low login rates but but again the norm the guidance district-wide was online learning but teachers were given the freedom to use paper and pencil and so there was some sporadic paper on pencil usage um as far as giving out one-to-one devices this district did make attempts to do both uh to deliver one-to-one devices and to deliver uh uh wi-fi uh or other sort of ways of accessing the internet for students that didn't have it um we never we we never pin down or they never pin down the actual rates uh of those things like uh some of our partners in ohio for example were able to say like you know we got to every single household or we got to all but like 12 households and our smaller partners can save things like that this district is a little bigger and never quite was able to pin that down um but they did get pretty close to one one device per household and they were finding ways to make wi-fi available throughout the city um one thing i'll note though is that i'll show a trend trend slide in a second um as they were rolling these things out the rates of online participation were not going up so one of the interesting things you see over a four week period that we're showing here is that by the end of that four week period a lot of the devices had been delivered a lot of that those wi-fi hotspots had been distributed and you're not seeing any upticks in actual login rates and that obviously generates all kinds of hypotheses about what's driving the uh the login or non-login um but the hypothesis that it was a lack of ability to you know a lack of a device or a lack of access to the internet um it's a little bit a little bit weakened by the fact that we didn't see any change in rates over time so i'll go to those slides in a little bit any other questions not at this point thanks everyone for adding those questions to the chat box yeah thanks so much great questions uh so next i'm just going to jump into uh who's logging in okay so this is sort of the more granular student level look by student characteristic of who's logging in to see if there's any patterns that the district should either be concerned about or or at the very least that the district could act on um and i'll note here uh that there are some jarring patterns the nature the the patterns themselves were not sort of jarring to these districts like were terribly surprising i should say but the degree uh the the degree of difference between subgroups uh was definitely disturbing to the district uh and so they've started doing we'll talk about this in a little bit but they are using this data to sort of draft their plans for this upcoming year and do some targeted outreach based on what they're seeing um but but it's not that the patterns are surprising it's that the degree of difference is surprising so starting with students eligible for a free and reduced price lunch uh we see here uh that they were less likely to log in uh that's about 12 percentage points less likely to log in um and there's sort of two things notable here uh in addition to this 12 percentage point gap uh one is that um this is these are rates of log in across an entire four week span which means even amongst students not eligible for a free and reduced price lunch nine percent of students didn't log in a single time in four weeks um and on the eligible side uh that number is far far scarier which is one in five students didn't log in a single time uh in in a four week window um and that is way way way different than chronic absenteeism right the pattern's basically the same right there's uh students eligible for free and reduced price lunch generally miss a little bit more school uh sometimes a lot more school but you don't see one in five students missing uh an entire four-week span like this so so again the degree is the scary thing here that for the district um the areas of the city with higher concentrations of students eligible for green reduced price launch generally had those lower log in rates um so this again is just trying to dive into some hypotheses that were generated by looking at that initial uh map of the the city's login rates obviously one hypothesis was those higher concentration areas those areas that are that have purple uh a lot of purple which is a high share of students not logging in uh were also areas that had high rates of pre-introduced price lunch and nothing in these charts really disabuses uh just abuses of that hypothesis um but again this has serious implications for how districts are targeting their outreach efforts uh we see a similar thing for students with ieps again students with ieps are very very likely to not have logged in at all uh there's a little bit more of a qualifier here uh on this data which is uh i don't know who said it because i can't see the chat window but the point about um is there paper and pencil going on uh it is very possible that there are is a greater rate of paper and pencil usage for students with ieps especially those students of ieps that would make sort of independent or virtual learning difficult um and so again it's not terribly surprising to see students with ieps uh having being less likely to log in than students without them uh but this is nevertheless a sort of jarring number which is almost one in three students with an iep never logged in to any platform at all um so i'm gonna pause there uh and i'll note uh as i pause here uh for questions in the chat window i'll note that uh we have more detailed slides on all of these things they're living in the appendix just in the interest of time i didn't want to kind of waste your time going through every analysis we did um but we have a whole appendix of slides and if somebody's really really interested in a few other things that we did we sort of have a whole whole catalog of additional slides that we generated for this district as well but uh but for now we're going through sort of the high level takeaways for you great thank you dave um there are no additional questions in the chat box at the moment and just to let you know lisa has been um following up with the questions about the sample size so thank you again lisa for um providing that information in the chat awesome thanks so much lisa um so this is a a similar thread but one of the things that we wanted to see after seeing that there were these kind of very obvious gaps between uh different groups of students wanted to see if those gaps were persistent or if over the four weeks span you know students started to catch up or get a little bit further off and it looks like those gaps are basically persistent which is to say the gaps remain roughly the same size almost no matter how we measured it uh over the entire four week window uh so the average number of days logged in per week um by whether or not a student was eligible for a free introduced price lunch um stayed you know roughly one day less uh one whole day per week less across the entire time span um but the other thing i'll note is is back to what i had uh alluded to before uh that even though the district was ramping up efforts to give folks access uh or to address issues of access uh over this four week window you're actually seeing the number of days a week students were logging in decline not in pantries um these are really small differences so we wouldn't you know we wouldn't necessarily jump out of this and say that decline is huge kids are really stopping engaging um but it's directionally enough for us to say they're not dramatically increasing and engaging either right that's very very unlikely from this data um the uh i'm gonna dive into the comparison between chronic absence and uh and login rates um but this is sort of a preview of that showing that these gaps are actually astronomical if you look at students who are previously chronically absent uh your average student who is chronically trending chronically absent before covet 19 logged in roughly once a week uh on average um and i'll show you in a second but that's that average is brought down by obviously a fair share of students who didn't log in at all um but even the students who were logging in weren't logging in at these like really high rates so you're talking about you know students who even on the on the high end were missing one in five days of school 20 you know double chronically absent 20 absence rate are missing or not doing any kind of uh educational engagement of any kind for four days out of every five right you're talking about a very dramatic increase in uh in uh disengagement um and again these patterns are just very persistent right we're not seeing any kind of gap closing or any changes over time that would suggest that this got better as folks got used to virtual learning uh so i'll just pause again see if there's any questions before diving into the the absences um we do have some questions uh so i'll i'll read you um nick's comment here um these analytics are all really helpful for us to think about measuring a minimum bar participation have you had any thoughts on how you would try to use any of this data to tackle active engagement in learning how do we start to differentiate those who just log in and maybe walk away from their computer versus those who log in and are active in the platforms great question that is a great question and worth way more than 20 minutes we have left to discuss but um but that is literally the whole purpose of this right answering that question is why we're doing the analysis in the first place and so the short answer is we've definitely and our partners uh our partners especially are giving a thought to exactly that question um we have no you know no concrete nothing concrete developed on this yet i think we're going to be uh you know getting into more of the data from this upcoming year uh before we can really figure out exactly how to generate analyses that that drive those deeper dives um that said uh we have a couple of considerations around this that are that are useful one is uh single sign-on platforms uh are not great at helping answer the question you just asked and i think the name was nick if i'm not mistaken um those the single sign-on platform tells us when a student logged in not how long they logged in for not what specific platform within it that they logged into it is a very limited data set that is the functional equivalent of telling you that the kid opened the door to the school right it's not telling you which classes the kid went to it's not telling you that the kid uh you know stayed all day um it's literally capturing the same thing as if a kid walked into school opened the door the door captured that he walked in turned around and walked out that would look exactly the same in this data as if the kid walked in the door sat and engaged in every single one of their classes and so so platforms like this are not going to answer those questions what they're going to do is tee up those questions and then you're going to need to find additional data sources to dive deeper into that um something like a learning management system will be much much better at questions like that right a learning management system is going to capture not just whether a student logged in at all although it will catch for that uh it'll also in most cases you know canvas schoology sort of these leading ones will generally give you reports about the types of interactions students have within the learning management system are they submitting assignments um are they uh you know are they if they have multiple courses within their learning management system are they addressing each of those courses are they checking all those things so you'll start to be able to get uh with learning management systems and things like that better data than what we have right now um and then there's this kind of third part of the question is even with the better data how do you address it once you've learned or at least have a really strong hypothesis that students just aren't actually engaging much right what are those um like what are those things you can start doing to address this hypothesis that uh even if they're logging in or or if they're not logging in that students just aren't engaging in their learning and we should be seriously worried about this um and that's uh that's where sort of these processes that we've developed over time with early warning systems come in um but for anybody that's familiar with how early warning works um you use the early warning system not necessarily to diagnose exactly what's happening or why but to identify the set of students with who for whom you it's most urgent to be able to answer those questions and then to develop a strategy for answering those questions for that specific group of students right um and we envision using this kind of thing exactly the same way uh and it's basically the same way we've always used attendance with our partners um we we do diagnostics on attendance to identify sort of the share of students and which students aren't showing up in school at all uh and then we dive deeper with that set of students to develop interventions that are customized to address the reasons that those students aren't or aren't showing up in school we would envision doing literally the same thing in fact we are envisioning doing exactly the same thing with our partners this upcoming year um i know that's probably not the most most satisfying answer a lot of it's a lot of sort of we we are going to have to work together to figure this out um but that's uh at this moment that's that's the world we're in uh was there any any follow-up questions to that all right thank you dave uh no there are not uh follow-up questions at this point great um so one of the things that is is sort of disappointing but also encouraging about our ability to use this data in the absence of um of the the physical in-person attendance that we've always had uh is that the patterns we we found that the patterns of uh login and the patterns of ads in prior absenteeism are sufficiently similar that even though we don't exactly know what it means when a student log in logs in um again to next question even though we don't know how long the student was logging in we don't know how long or what they were doing exactly when they logged in um we have the patterns suggest that uh there's there may be a similar level of sort of early warning canary in the coal mine uh sort of predictability of not logging in at all to not showing up in school uh so high absence is highly correlated with not logging in uh so we started geographically like we did uh throughout the process um and what you can see here is that the share of students not logging in has high concentrations around the city in a very very similar pattern to the share of students uh sorry the average number of absences per student and this this chart we don't have like the 40 different iterations of these charts here for today but this chart looks very similar if you just did the percentage of students chronically absent prior to uh through march 15th when this will shut down um so preliminarily the hypothesis that patterns of absenteeism uh sorry patterns of login reflect patterns of attendance uh this data does nothing to sort of change that hypothesis um this is the side though that really drives home the risk that we face uh in a virtual world of sort of losing touch altogether with these students who even when they were chronically absent we still had contact with on a you know mostly more often than not again a student's chronically absent you know misses ten percent of days shows up one you know misses one day out of every 10. right so nine days out of every two weeks you get to put eyes on that student you get to engage with them you get to you know even if they're not fully engaged in their instruction you see if they're okay things like that and we're not seeing that equivalent uh anywhere near that equivalent in uh in the remote setting so what you're seeing here is that for students who are chronically absent before uh the school shutdown on march 15 in a four week window fully one in three never logged in a single time and for the most part especially if you're in a larger district uh trying to do sort of one-on-one outreach over a four-week window to catch up with all of these students just to make sure you can kind of put eyes on them and make sure everything's okay um is very very difficult to do right um especially if you don't have perfect phone numbers before covet if you don't have addresses and this is one of the things that attendance works has been really hammering about is we really need those phone numbers and addresses now more than ever and this is the reason is those natural opportunities to connect with us your most at-risk students um are not arising in a virtual setting in the same way uh and so again are we surprised or was the district surprised that there was a gap between chronically absent students and not chronically absent students when it came to logging in of course not um but a 22 percentage point difference in in another's 22 uh percentage points more of students never logging in a single time in four weeks um that's a jarring difference right that's a really big a really big number uh and so that's that's driving a lot of sort of uh continuous improvement thinking on this district's part about right what's causing this how do we target it how do we address it um neighborhoods with higher absence rates uh so this is just a sort of correlational look at the same thing um but what we found was that uh not terribly surprisingly that the neighborhoods that had the higher absence rates had a higher percent of students who hadn't logged in and the most important takeaway here is look how straight that line is right the the predictability of prior absence rate or sorry how how well prior absence rate predicts whether students logging in is reflected in that like very very clear straight line that you see here and we have sort of the the flip side of that um which is uh the this the neighborhoods that have students with higher absence had lower overall rates of login uh so these are slightly different measures of the same thing and all it does is sort of flip the flip the chart um but again you see very very clear these dots are tightly clustered around a very straight line all of which suggests that um we to the extent that we can use absenteeism to predict all these other things that we care about it seems that we can use absences to de facto predict absenteeism i'm sorry uh logins to predict absenteeism and if that those hold as tightly as they may they appear to hear um there's at least reason to be optimistic that virtual logins are uh are going to be a strong early warning indicator once we actually start to be able to do those kinds of analysis directly so i'm going to pause there and see if there's any questions about the relationship between absenteeism prior absenteeism and logistics there are no questions at this point and we've got about 10 minutes left in our time together um i just wanna if you do have questions continue to put them in the chat box but we'll also have opportunities for a follow-up as dave alluded to before so if something comes to you later after um we'll make sure we get those questions answered uh so i'm gonna do a quick review of the data sources uh nothing nothing crazy or groundbreaking here these are all data sources that uh that folks will will have access to um so as i noted the data we got was a a single export from a single sign-on platform called clever uh and that clever platform was designed to provide access to sort of single single sign-on access to digital resources um the um the data received as i noted is literally just a time stamp of when students logged in um it is doesn't tell you duration of login it doesn't tell you what platform within clever that they logged into anything like that time period was four weeks um and uh again as i alluded to um to the extent that they were doing using other online platforms like google classroom or zoom which don't log in through clever um we're not this wouldn't capture that and so you could imagine sort of systematically under counting uh participation uh if you were in a district that was using multiple platforms and you only pulled the data from one um i'll note that google classroom is not terribly easy to pull the data from so if you wanted to pull it from two platforms one being clever one into a classroom and merge those together you could uh it would be a little tedious but there is a free provider whose me tried to make things like that easier um and zoom is the same way it's not the easiest thing to pull participation out of but it is technically possible um so this all highlights that uh and i think this is very consistent with the guidance od he gave on tracking attendance but having a a sort of consistent platform throughout the district that is going to be the keeper of sort of the analog to physical attendance the keeper of did the student have in any way it was was the student in any way exposed to their educational opportunities uh on a given day is really important right so we strongly recommend sort of a district-wide learning management system or some equivalent to that where every student has an expectation of logging into that learning management system in the same way we have an expectation that every student show up physically in a building right now and so uh we very very strongly recommend that it'll make it much much easier for districts to do the type of uh sort of intervention targeting informative uh data analysis that you need it's not impossible if you don't do that but it is very very different the geographic data we got is we start with student addresses straight out of the student information system so our districts send us a file that has all the student info including their address um we then uh use geocoding data from the center for geographic analysis at harvard um the sort of the the geocoding data is not itself something that has a cost like you can you can get that from anywhere um it's sort of the the effort of taking an address using geocoded information and combining those two two things uh and we'll talk about that in a little bit um for the demographic data we use the student attributes file uh and a student school year file uh and again these are both just ordinary polls out of the student information system um we did this all out of data that districts would be sending anyway or using any way out of their student information systems and then the attendance that is again just the ordinary daily attendance file that you can extract from pretty much every student information system districts might use um and i'll also note that even though this was not necessarily ohio district that we did this for um all of the data files we used for this we get from all of our ohio districts uh everything except clever because our ohio research wasn't using clever in the same way but um but all the other kind of background data the attendance data um the student demographic files the attendance files those are all we get those all in a way that we could use for this from all of our hybrids um i'm just going to combine these but obviously heather don't be shy about hollering at me if there's any questions as i go through but i do want to make sure we cover this before the time ends um yeah we do have one question that i'm gonna i'm gonna share with you i feel like it's gonna be a longer answer so we might have to come back to it and share information afterwards but there's a great question about have you found any successful interventions based on this data so you can um think about that one while you're talking about methods and tools and just wanted you to know that that's up there awesome so it's actually a short answer right now the direct version of that which is not yet so we are even though our pilot cycle is very very short and only takes you know somewhere around eight weeks at the short end twelve weeks at the long end to figure out what works we haven't had anywhere near enough time to do that uh with this district or any of the districts we've done this kind of thing with uh so so the short answer is we have not yet actually found anything that doesn't we have all kinds of hypotheses and we can talk about those but we do not have any kind of concrete evidence on this yet um so the the specific methods um so we start by g ocoding the student data as i noted and that involves sort of your basic cleaning of addresses literally just making sure they look like addresses because sometimes they're open fields and sis you got to put the commas in place things like that um and then we use a geo geocoding program called arcgis um uh and that that does that is not a free tool um it is a really good one but it's not free uh i noted in the bottom uh in this there's a little asterisk down here um that uh there are free versions i'm literally just made up or gave an example of one called qgis it's totally open source and free um but uh and you can do everything that we've done in this with with sort of off-the-shelf free programs um the bigger issue is kind of knowing how to use them and all that and uh and if anybody is really interested in diving into that we can uh we can do some follow-up on that we create a final analysis file by merging all of the different files so we take the geographic file that includes all the geocoded data the demographic file the attendance file and the single sign-on data or learning management system data and we merge those um we use uh in-house uh for the merger i believe we used stata but you could use stata uh or r for this um or honestly as i put in the asterisks any other um any other platform um statistical platform will make this easy you can use sas if you use that event if you use spss you can use that they can all merge files there's nothing magical about how sata or r do it that can't be done by other platforms that said um um it would be very difficult to do this in something like excel it can be done um but it would be extremely painful potentially so uh so it is worth kind of figuring out how to use any of the other platforms to do it if you already know how awesome we do recommend just use whatever you know how to use don't learn anything uh if you are going to learn a new thing i personally recommend r simply because it's free free off the shelf um geographic analysis sort of making the the pretty pictures that rachel made with all the map the heat maps on them uh uses uh off-the-shelf uh uh packages in r uh we noted the names those packages in here if you're not comfortable or familiar with r those won't be terribly terribly uh noteworthy to you but uh if you use r and when you use r it's really really really clear how to download those packages and use them um the uh other pat you can do this in other things you do not need to use special r packages like i said if you're comfortable in stata or sas or spss you theoretically can do this uh we just happen to use r to do it um and then the descriptive analysis that's probably the least interesting thing of all um any any any package can do it or if you had a large uh if you somehow managed to get all this into a single excel file you could theoretically generate a lot of these graphs and stuff uh in excel um so in our last minute i just want to kind of gloss over these these considerations um so i noted this already but how complete your data is is really critical if only a third of your students are supposed to use clever it wouldn't be surprising to find out that 20 of your students uh are only 20 of your students logged in um you know you need to make sure you know the denominator when you're doing this analysis and the easiest thing in the world is if you have sort of a district-wide mandate to use one one platform um do you understand the coding of the data that you're getting uh so it's one thing that to pull data it's another thing to know what that data says um and you know we've we've started working with some districts assignment data uh which is you know you'd hope would say assignment completed assignment not completed but it turns out the coding is a little more complicated than that and without going back to the district we actually didn't know exactly what all the numbers in the file meant so you know understanding the code in your data and making sure they're consistent is very helpful um but the most important thing potentially of all once you actually fundamentally have the data available and know how to process it um is making sure that you're generating analyses that the districts can use and so what we're really trying to do is generate actionable analysis what is the what is the finding and how is the district going to use that finding to serve students in fact um and finally if you can't do it relatively routinely it's not going to be super useful so if it takes you three weeks to prepare analysis every time you have to do this uh four weeks and you can't sort of set it up so that's reasonably automated it's gonna be very hard to use it actionably you're not gonna be able to see changes in patterns quickly enough uh things like that and so another main consideration is can you do it routinely can you spin this analysis up and then sort of replicate it relatively quickly um so i know we're at time thank you so much for being bearing with this uh i have a few extra minutes so i'm happy to stay on um uh and answer any questions that are already in the chat window or any new ones that come up i can stay on for a little while longer um but uh but for those of you who are like having to jump off right now i just want to say thank you thank you for listening to this thanks for engaging and asking questions please don't be shy about reaching out uh you can reach out through ode uh you can reach out directly to us we do have twitter as i noted so that's a great way to reach out to us too um and uh we do have a very very brief survey shouldn't take more than three or four minutes to fill out if you have the time uh and can spare it we would love to get your feedback on this we do use that to sort of uh drive drive the work we do and and try to improve these things going forward so uh so if you can take a minute or two or three uh to fill out that survey we very much appreciate it wonderful thank you so much dave uh just want to say thank you to dave to rachel lee and to lisa for all the work that they put into preparing these analyses and this presentation this morning and also just say again thank you to all of you for joining us this morning i hope it was helpful and interesting and generated some new ideas and i look forward to hearing those ideas and engaging in this conversation um over the the year to come because we know it's a complex topic and we're all going to continue to learn together about how best to look at this information and use this information and so um this is a drop in the bucket of the kinds of conversations that we can have about this topic um so thank you so much and we will as i said make this recording available to all of you and share the slides and if you have questions after today please feel free to follow up either with proving ground or with the department and with that um we don't have questions at this moment dave but we can we can certainly wait a little bit and make sure we catch everything while we wait i can dive into a little more depth than that the answer i answered really shortly about other interventions that uh that work to address these things um the the short answer again is is we don't know yet um we haven't done with our partners and to our knowledge we haven't seen anybody else do a sort of rigorous evaluation of any any of these efforts that have taken place since since will shut down for coping um we do however have a lot of knowledge we've generated pre-covered that we have strong hypotheses uh would uh generate lessons that apply uh in a virtual world uh things like um you know we have we've identified small behavioral nudges that at relatively low cost and effort on the district's part do seem to actually improve attendance um things like sending personalized robo calls every six weeks we saw three to four percent in some cases up to ten percent uh improvements uh uh reductions in absences uh based on simply setting up those automated robocalls um as long as they were personalized and while we don't know if we'd see anywhere near the same type of effects uh or say virtual learning let's say we send robo districts and robocalls trying to popu get students to log into an online platform um we have no way of knowing if we get the same results but we do but we can hypothesize based on the root causes that those robocalls were addressing that uh that it's worth trying uh and systematically figuring out and uh because those root causes may be the same right if a student is not if a younger student's not showing up because their parents not aware of how much how important it is that they show up or how often that they were actually showing up or the impacts of a given day of not showing up um they may be not logging their kid in for the same reason they're sitting in for the same reason and so uh so there's reason to believe uh that at least for some parents those robocalls would be would be affected um so we have lots of hypotheses like that um but i have no no hard or rigorous evidence either way it'll be interesting to see though how that all unfolds is you're able to see more data and that's what i was thinking about as you were showing the the weekly data points um and thinking about how that's really interesting to look at from just a descriptive perspective but also that those are the data points that will be um most valuable as districts start to make changes and you kind of alluded to that when you talked about you know they they distributed the one-to-one devices and it really didn't seem to have a significant effect as they looked at the information over time so um that'll be really fascinating yeah yeah very much looking looking forward to having sort of more comprehensive data on this kind of stuff um lori noted that several districts are planning parent trainings this fall to engage and inform parents with online platforms um so that parents feel more empowered and you know aren't sure we're not sure if that will make an impact but gonna try it and see if that works and that's that's what a lot of everyone is going to be doing this year right iterating and learning um and adjusting as we go absolutely all right well i don't see new questions here but thank you dave very much and um brittany thank you too i know you're still online thank you very much for helping to monitor everything and keep things running smoothly absolutely the perfect end to a covered webinar absolutely absolutely so appropriate um all right well thank you everyone and have a wonderful rest of your day you