eSigning Summer Camp Evaluation Made Easy

Esigning summer camp evaluation

and realized I welcome to session 3 of the free online DML Commons course program evaluations for connected learning I'm bill Penn Ewell and this is a course where you'll gain practical experience in the process of educational program evaluation by exploring real-world problems associated with case studies of connected learning today's guests I are from the few studios in Chicago Illinois fuse is an interest driven learning experience developed by researchers and educators at the school of education and social policy at Northwestern University and the program engages preteens and teens and steam topics while fostering the development of important 21st century skills fuse is based on extensive research and interest development and interest driven learning and a reminder to everyone watching today you can use the hashtag CL eval to pose questions on twitter or use the chat feature where you're watching this live web cast i hope you'll pose questions of our guests today in this way and in future sessions so by way of introduction I want to introduce first dr. Kenny Jonah who is a research professor at Northwestern University who leads projects in cyber learning computational thinking and blended learning and new games based approaches to engaging youth and interest driven steam learning making and tinkering he's worked to the forefront of the learning sciences and learning technologies fields for over 20 years he's a thought leader and explain plying insights from cognitive and learning sciences to the design of technology supported learning environments particularly in areas of STEM education i also have from the fuse program Maggie Waldron is the program director and fuses a program as such as we've mentioned that engages students and it's team science steam projects that are interest Riven ages 8 11 through 18 she's also a member of the office of education partnerships at Northwestern where she's developed research-based science curricula and led numerous professional development workshops for teachers prior to joining the teams at fuse in OSAP he worked as a research assistant at Palmer station in Antarctica studying the impacts of climate change on the Western Antarctic Peninsula Maggie holds a BA in biology and environmental studies from Lawrence University and will complete an MA in science communication from Texas Tech University in December 2016 and Daniela cruella de Jamaat giacomo is a PhD student at the University of Colorado at Boulder here in our School of Education Department of learning sciences and human development program she's working as a research assistant with Professor Chris Gutierrez and the connected learning research networks project on leveraging horizontal expertise and she's integrated that and she's very interested in the design of learning spaces for non-dominant youth specifically Latino immigrant communities in low-income families prior to her pursuing her PhD she worked in the Bay Area and hybrid secondary schools Mental Health and Social Work agencies and political asylum and refugee rights organizations she holds a master's and Development Studies from Cambridge University and a bachelor's in Latin America and studies from UC Berkeley so welcome chemie Maggie and Daniela to today's session so I wanted to start with Cammy and Maggie and just ask you all to say a little bit about some of the goals that you have for a few studios and also then to reflect on what some of the goals that the young people bring to the table great so I'll start with that one and cami will jump in if I leave anything out so we created the fuse program because we wanted to create a something for students who are not already very high performing in stanmore steam or who are not confident in those areas we wanted to create a program that would be very welcoming to them and a place where they can take risks in those areas in a comfortable environment so that was our primary goal in starting fuse and that was about four and a half years ago and and that goal has continued to drive the way that we think about activities and the whole space when we do our Peter professional development as well as the way we design the web platform that facilitates the fused experience so I'm curious too so what are the some of the agendas that your young people bring and and how might they be different for example I know you all are both schools and after school settings you know how do they think you know what do you think they're thinking about when they come to a few studio I think that it probably varies quite a bit you know during in the in school sessions so fuse now I'd say about 85 to 90 percent of our youth are doing fuse during the school day which means it's not always elective for them to be infused it might be something that everyone in their grade is doing for example in their school so the first couple of days the first day they go to fuse they may not have an agenda you know they may not know what they're being invited to participate in but I think that once once they've started one of our challenges one of our activities of use the their goal my sense is their goal is is to keep exploring to keep working on what they're doing to work with their friends and you know to really kind of yeah I would say to continue to explore through fuse in the after-school sessions where participation is more elective I think there's probably a lot of diversity there as well we know that a lot of times students come because one of their teachers has encouraged them to you know they suppose need a little bit of a nudge or one of their friends has said hey you should come with me today and again I think it's hard for them to know exactly what they're getting into until they really start the first day but I think what makes kids keep coming back as the their ability to continue to explore things that really mean something to them having anything to add from your point of view about the goals that you have or that young people bring oh well those are kind of two different questions I mean I guess what I would say to amplify Maggie's points and just to make it clear um we designed fuse to really focus primarily on interest devel and especially that the very earliest stages of kids going from either i'm not interested in these topics at all or maybe i'm not even really that into school at all to becoming discovering and becoming interested in something and i think that's a pretty fundamental shift it's not that we're not focused on learning because we very much are obviously we're learning scientists and we're all about kids learning but i think in general i think we feel that the ideas of interest development have really taken a backseat to learning and infuse we want to put them at least in the same seat if not in the front seat with respect to learning and so you know our goals as researchers are to understand how do we support kids in becoming interested and then sustaining that interest over time and learning along the way in developing other important skills and abilities but you know as program designers and program developers we we want to make sure that every kid who walks into a few studio finds at least one thing that they can get interested in a long and a quick follow-up what what are your key strategies for actually doing that to help them discover an interest that maybe they didn't know that they had or or help them develop a new one yeah I mean there are several kind of key components and I think we can continue to unpack these throughout the session today but I would say first and foremost is the idea that it's fuses really a interest driven or choice based format which ultimately means that the students or the participants are free to choose to work on or explore any of our two dozen or so challenges there's no adult direction in terms of what they're supposed to do when they're free to stick with a challenge as long as they want they're free to quit a challenge they're free to choose whether to work by themselves or with their friends they're free to switch that arrangement dynamically along the way so so at the core of it is this idea of choice of interest driven exploration and I think the complement to that is that fuse creates a safe environment to try things into quote-unquote fail without any of the normal kind of consequences that traditional schooling or even other programs frankly out of school often explicitly or implicitly apply so if you want to foster interest driven learning you have to obviously create another enough opportunities to find something you're interested in but you also have to kind of make it very very okay to try things and then quit them if you're not interested in them and that quitting can't be happy can't have a negative connotation that it typically does in other educational contexts any thanks so Maggie and can you tell us a little bit about the places where fuse is being implemented and what some of the different models that are being used for implementation and schools in the after-school sites sure so we have a lot of different ways that fuse has been has been implemented over the last several years and that's one of I think the most exciting things for us is to see how flexible it's been and continues to be so we have about we've over 60 total sites across the Chicagoland area as well as in California and Ohio more than 50 of those are school-based locations and so sometimes that means fuse is running as an after-school program sometimes it means most often it means that fuses run during the school day and that can be an elective that middle school or high school if students are taking it could be actually a core class that's offered to use the word classes a little bit sounds little more formal than we really want it to be with but that's kind of how how the schools under 4 perceive it so it might be something that every 5th and 6th grader will do over the course of a week the school is allocated maybe 90 minutes of time by carving out minutes from different other periods to make it something for everyone so those are that's kind of a high-level overview of what it looks like at schools we also have a number of out of school in back minutes youth centers or libraries that are running fuse maybe on the weekends are in the after school setting and so when you think about some of the ways that site you know sites very right and their implementation and one of the things that evaluators need to pay close attention to is that variability and implementation and in some of those I imagine you're seeing some enhancements some things that maybe are making the program better because they're fitting into a specific need that you hadn't anticipated and then other ones were a little worried and maybe you're thinking wow our program models at risk could you speak to those both the enhancements and your your worries about adaptations that are putting the program integrity at risk yes I'm just trying to think of a good example so I think one of the one of the things we've really enjoyed seeing is is how teachers who are what we call facilitators in this space are adding to the few space by by asking a little bit more of their student or the youth there to contribute in terms of reflecting on their work and that sounds one that we built as a neccessity into fuse because as companies alluded to we wanted this to be a very youth centered free choice self-paced type environment and we didn't want to add too many layers of other things onto it but some teachers have really successfully done that in a way that doesn't interfere with I think what is the core experience and so that's been really great for us to see and I think we've really learned a lot from that yeah I just just to amplify that a little bit bill on and I think one of the challenges that evaluators face in these kinds of program models and fuses specifically is is that we very intentionally created a model that is what we call half-baked and in the most kind of positive sense of that term as opposed to kind of the usual negative sense of the term and we have baked it very intentionally because we wanted it to be taken up and adapted in the broad a kind of variety of settings both in and out of school libraries community centers summer camps on and of course you know it turns out that that half baking as you alluded to has advantages and risks carries risks as well and so we've been very pleased with the kind of half-baked pneus of it because it has allowed schools to implement it in a wide variety of ways of MAG you was saying both during the school day as a required course as an after-school club and in libraries but it does carry risks that teachers or others will sort of either intentionally or inadvertently impose things on the model that that would harm the I hate to use word fidelity but would harm the intent or goals of the model so the easiest of one is the idea of say grading participation in fuse in a variety of ways and we're constantly engaging with our school partners in a discussion about what assessment could look like or should look like because again as I said earlier one of the key things that makes you successful is that it is a safe space to explore to try new interest to you know fail at a challenge and try again and if you know that you're being graded that pretty much eliminates a big chunk of that safe space and so you know the half baked I think is a great approach but is one that you have to constantly be engaging with your partners on and and Maggie and the team do a great job of sort of working through those discussions to try to balance the goals of our partners with the design goals of use what about choice alone because I could see a school requiring certain challenges for example have you run into that or and if not what have been your strategies to maybe keep that from happening sure now you want to talk about that sure so sometimes there are there are ways that schools limit choice and it we don't necessarily think that that's a bad thing so for example one of our partners where fuse is running actually for sixth seventh and eighth graders across the district they've subsets of challenges so the students had choice within a certain group and then they always have something new to look forward to in the in the coming years and so that was something that was new with that partner but i think has worked really well and continued to have so kids are always excited about something to look forward to we've had other partners who as you know to build up for coming was saying earlier knowing that they have constraints in their district that they need to be aware of standards and they might have grading involved they have sought to limit choice in order to try to align challenges or the challenges that kids might be experiencing to certain standards and so that's been something that we've really had we've talked with our partners a lot about to go back again to what cami was saying and kind of returning to initial conversations about why they wanted to implement fuse in the first place which I think a lot of schools are drawn to fuse because of its ability or it's an opportunity to kind of promote a lot of the 21st century skills that that we know are so important and so if we return to that as it as the reason for implementing fuse I think it's easier than to for everyone to be more comfortable with the choice that is available and not feel like they need to restrict it to certain ones that might align better with you know science or engineering standards and I wonder too if you run into attention between incurring to encourage students to go deep and to level up as opposed to really try to explore lots of varied interests and and how do you work with that as a program director so we try to encourage or we talk a lot about the different ways that that there is value in the different types of experiences that kids have it fuse so you know that it's important to to value and congratulate the the youth who levels up through six challenges in a row you know completes everything and really goes all out and it's just as important to acknowledge that even though one student might be spending months on at one particular challenge you know that that's also there's a lot of value in that and they're they're gaining so much from that but also this idea of you know dabbling it's kind of what we've been calling it kind of jumping around from level 1 to level one across the set of challenges you know I think that we've really tried to make it clear to our facilitators that we see that as being valuable to something that some of our schools have done to kind of take that a step further is to ask that students who start a challenge I start a level complete it so that way that they're they're not you know just being it gets hard they don't just give up but they you know they persist a little bit through because sometimes there's just a little bit of a hump right and having have some having encouragement from up here from a facilitator in the room to get over that I think is something that a lot of our facilitators have have said is helpful one quick follow-up oh kami you're on it yeah I guess I guess what I was going to add to that bill is I think the diversity of engagement patterns or even if what we call may be acceptable engagement patterns you know that universe is by by intent and by design very broadly defined infused and I think that poses challenges not just for our school partners but from an evaluation standpoint as well because you know when our external evaluator walks in you know they're going to be wondering like how do I make sense of whether this thing is working or not what you know what do I look for among kids and their behaviors and stuff and so immediately you know that becomes an issue there as well because you have to be flexible enough to have a broader definition but frankly I see it as an opportunity in and one of the things that we've seen with our teachers and other facilitator partners is that engaging with them in this idea that there are multiple ways to achieve success or to engage successfully from a youth perspective is eye-opening and in fact educational for them and helps them peel back a lot of the kind of preconceived notions that kind of school based assessments impose and I think often when we're doing connected learning programs broadly speaking on all of us sort of deal with the legacy of kind of a school-based prism through which many people you know view assessment of view evaluation and to kind of want to count things the kids are doing and very kind of limited in my opik sort of ways and and so we're trying to really push the push back on that idea and and give facilitators whether they be teachers or other youth workers a bigger idea richer idea of how to understand and appreciate learning and engagement in all of its diverse forms that's great and in a few minutes I'll give Danielle a chance to respond to how the fuse evaluation is tackling that issue but before we turn to that this is a unique project and it is one that is true of many NSF projects there's a research team and there's an evaluation team tell us a little bit about the research that you're doing on the fuse program at Northwestern if you could please and also the platforms that you're using to do that so the fuse platform itself I think is a is a context where you're gathering data right absolutely so I mean at the highest possible level you know we're asking very kind of fundamental questions about interest development and engagement as I as I mentioned earlier we're very focused on those earliest stages of taking a young person from not interested or not sure or even actively afraid or resisting stem or steam ideas and topics to being interested and engaged in that and we want to follow that in this context that we call few studio which is basically a room full of kids that are working on fuse together engaged together and so you know the questions are for you know from the basic sort of like what do kids learn as they participate in fuse and engage in the different challenges what how do they work together how do they form different learning arrangements dynamically through their participation one of our grad students Lauren penny is working on her dissertation looking very specifically at the development of what we call relative expertise how do kids go from kind of not knowing very much to knowing a little bit more than some of their peers and then how does that relative expertise actually facilitate kids up here helping and peer collaboration who do kids turn to for help how do they recognize up here that has that expertise or conversely maybe they you know don't have enough expertise and then they turn to somebody else and we're actually seeing really interesting evidence of that merge we're interested also in how do the skills and practices that kids pick up in few studios how does that extend into other parts of the school day and then other parts of their lives outside of school as well both in terms of new interest that they're pursuing as well as you know maybe new elective stem courses that they're participating in or even just ways that their teachers might be seeing them work together differently now that they have been infused and we're seeing evidence of all of those kinds of things emerge to your point you know the other interesting thing about fuses were pushing the boundaries of marrying both quantitative and qualitative data you know often it's either one or the other and infuse the fuse website mediates all the activity in fuse so students whenever they want to start a challenge have to go log on to the website click into a challenge and actually click on a level and say I'm starting this level and then when they complete their their challenge they have to upload a completion artifact which is a picture or video and then upload that to say you know this is my proof that I'm completing this level and so all that data is available through the website and so we also have demographic breakdowns by gender and race ethnicity so we're able to sort of slice and dice the data in aggregate but that only tells us that something's happening it doesn't tell us what is happening and so as Maggie said we might see a young person who sticks with one challenge for for six months straight and we've actually seen a young young person do this in our e-text I'll challenge and so the question is you know what's going on there we can look at the web data and see oh well we know that they've worked on the same challenge level for six months are they just screwing around or are they really doing something but that's when our ethnographic data kicks in and so we have a whole room video cameras that the grad students and the research team run as well as some really cool new visor cams that which are literally like tennis visors with a GoPro attached right underneath there that give us really amazing point of view video of what the kids are working on on the table what they're looking at on the screen if they turn their head and talk to their friend next to them we can see that as well and so all of these data sources come together through for analysis and for evaluation purposes and what are you hoping to learn different from the evaluation than what you're learning as a research team well we'd like to know some of the things that we're not looking for and I think that's a great role for evaluators to play but in general we and one of the pieces i think i forgot to mention and i hope danielle and that you can elaborate on this we also have a survey mechanism that gets popped up and hopefully that will get described a bit more later on to your point about not interrupting the flow but again you know when we're doing ethnographic data you know at most and we have a pretty big research team i mean we have you know three or four grad students and post doc but we can only be in one or two classrooms at a time and even then analyzing all that video data is incredibly laborious and time intensive as anybody who's done video analysis knows and so even when we're in a particular classroom and we're watching one student or were taking field notes our field of vision if you will is very narrow and so the evaluation I think helps give us a big picture framing of you know how are we moving the needle in aggregate across various a gender male versus female are we having you know equivalent impacts on attitudes and perceptions and so it gives us a you know the big picture view that I think often are our research team doesn't have because they're really going in-depth and doing sort of a micro analytic look at at learning and engagement great so Daniella what is going on and the evaluation right now for a fuse okay well hopefully this doesn't contradict what cami said too much but I do want to say that in doing the evaluation and as people who are much further removed from the site's themselves than the researchers the facilitators and the studios I think for us as an evaluation team it's important to keep in mind that the raw data so for example the data we have on how many students are completing particular levels how many students are completing challenges by leveling up what types of challenges are pursued more than others that that doesn't tell the whole story and in fact even if we understood the whole story and if we were there the whole time there are so many perspectives from which to make sense of that story and so I just before this was on a call with our are part of our evaluation team which includes computer scientists which computes learning scientist which can you know includes people like me that are struggling to identify with in any room you know I'm and for example allo allo completion rate on a challenge to someone that far removed though they're still involved in the team can mean something different very different than it means to Maggie ritsuka me or to the students themselves and so as you've heard the maghuin can be mentioned it's really important that the ways that we make sense of the data are informed to a large degree by not only ethnographic methods that are being carried out alongside the survey and the computer-generated information but that we're very cautious in the way that we present that data as well say a little bit more about this survey what you know what the fuse evaluation survey is measuring and yeah i'm actually i'm wondering if Maggie do you want to say a little bit about the the new survey that's being implemented and then I'll go back to evaluation sure uh the lights just went off in my office been sitting still too long um so we designed this survey together it was a group effort to augment the kinds of data that can be talked about earlier so we have in room observations in a small handful of our studio we have web data from everywhere and and so that even those together don't tell the whole story so through the survey we really sought to gather more information about what kinds of interests are youth bringing into the fuse studio environment the first time and then as they continue to come back and how do those interests change over time what are they then taking out and when they walk out of views what kinds of things are they exploring afterwards how has their experience at fuse influenced the way that they you know influence their different interests and how they pursued them or don't it also helps us understand the the perception of fuse and the fuse challenges by the youth who attend fuse so we have questions asking about know do you like them are they fun are they too hard are they too easy or simple things like that as well as who do you like to work on challenges with you know do you wait to do hard challenges with your friends or do you can you tackle anything you know really trying to get at those perceptions of themselves in the studio and you can be alluded to this how are these things administered these surveys yes so our surveys are administered through the website so in an effort to be to have as little interference with their experience at fuse we are doing the surveys and in light boxes when students are logged into their account on the website it's not a perfect I think we're still trying to figure out exactly how to do this because the fuses experience is so organic and so fluid but we're still trying to get that you know relatively reliable survey data so we have surveys that pop up on the website after certain time points depending on how long a student has been at fuse so you know their second day for a relative baseline several weeks later for some you know early time point and then again you know months months afterwards to try to see what kind of long-term effects that might be and say a little bit more because about the challenges that we've been working through in terms of implementing the surveys because this might be one of the challenges that someone who's taking the online course could actually help us would be great so let's see we we didn't want the surveys to take away from the momentum that a student might be having when they're working on it on a challenge so we tried to make them as short as possible that being said there's still there still lengthy especially if you're thinking about someone who's 12 years old 11 12 years old but so it was a challenge to pare down the survey enough that we felt like we were getting enough data to tell you know the rest of the story that we needed but also so that it wouldn't be too burdensome that's been one of our big challenges we don't a second challenge would be actually getting responses so we have tried a couple of different ways to encourage students to respond to the survey we tried to make it very very light-hearted and fun we haven't offered incentives but we do we do ask for help from the facilitators to encourage them to take the survey so in and with that support we've actually gotten much better response rates again not perfect but I think I think better than then last year that's great don't and Daniella chemie raised this issue of there's lots of different pathways lots of possible good pathways through fuse how is the evaluation team handling this like what how are they going about analyzing these pathways well I would say the first thing is we're going slowly and carefully and we were working closely with Maggie on the ground level to make sure that basically all of our understandings that there were very clear in the way we're making sense of the data as I mentioned and so for example this year in particular as megan kimmy have mentioned because fuse is taken up and experienced in so many different sites both in school-based non-school base elective non-elective for credit remediation or four for fun we're interested this year in the evaluation in getting a better sense of how the fuse implementation at bear cites impacts youths patterns of participation as well as their attitudes towards their participation the an interest in the activity themselves and of course particularly in regard to how they're making sense of steam oriented activities and dispositions and so though that I think I I do feel and I think bill would agree that it's we have such a nice opportunity to look at the way that a program is taken up and experienced all across the sites and so it would be nice to be able to tell a story of context variability in tandem with the data and that each side is producing um yes and can you speak a little bit this is a role that our computer science members of our team are playing say a little bit more about how what their role is and helping us understand these pathways yes well we are very grateful for our computer scientists on achievement we're looking and so what's nice about that is that and not only do we not only are we interested in seeing for example that changes over time so within a particular time that a student experiences fuse whether that's six weeks six months or over the course if they come back the next year and they elect into a fuse class them we want to look at how they're not only their participation but how their attitudes in response to the survey questions change over time and we'd also like to say something about the connected learning principles that are in cited in the survey so r is are the few studios creating the conditions in which youth feel like they can develop their interest that are peer supported that are academically oriented and is there an impact for example if students work in the same studio together are they working with if they work with their friends is that impacting the ways in which their relationship to the activity or to progressing through the activity shifts and so we're looking in that and calling it a sort of same room influences you know what ways is being in the same room potentially next to a student completing the same challenge may at a different level is that impacting the activity patterns of the youth and so it's very helpful to have a computer scientist to help to figure that out it's great and I know that the evaluation team is planning to study implementation in some of the sites and I'm different or maybe even better way to say it is how the different sites are baking fully baking the program from its half-baked state can you say a little bit more about what's planned there for this year yeah so this year in addition to being fortunate to have the three sort of baked in survey points along the course of a student's experience at fuse we're also going to be choosing and a collaborative effort with Maggie and the team at Northwestern nine to ten focal schools that really represent the extremes of context implementation so whether the schools are you are trying to move towards assessing fuse in a certain way whether it's elective whether it's mandatory and working with those program facilitators so the few site facilitators to ask them questions that from an ethnographic interview that will hopefully um tell us a little bit more about the ways in which they are making sense of how to provide the fuse experience and guide the students to diffuse experiences or not and how so the ways in which they engage with few is how that's mediating youth participation so hopefully I'm in tandem with not only the survey data the computer generated data from the activities as well as ethnographic interviews of the fuse facilitators we will be able to tell a little bit of a more complete picture and this is a question for everyone now evaluators and program leaders alike is what are some emerging questions for you in the evaluation that are coming out of the evaluation works so far or out of the research that you think are important for the program to be able to answer well I have I have one that comes to mind and then I see kami just unlike so I'll let him jump into you know I think one of the things that I think most about is is this diversity to implementation that Daniela talked about so more and more schools are coming to us and wanting to use fuse as part of their school day because of all the different skills and practices that kids can develop through their experiences with it and so I'm really interested in how how can we then meet that demand you know as schools have a lot of pressure on them to provide certain types of experiences for their for their students how can we how can we maintain what's great about few but also figure out how how to work with those within those demands so that's something that's really interesting to me I think we'll learn a lot about this spring actually before Kimmy asks his question how would you know if that was the case like what would you be feeling or observing you know that he said we're meeting the demands of this expanded group of sites so I think what I what comes into the forefront of my mind is when schools as coming it and I both talked about before when they feel these these constraints of needing to meet standards or needing to put grades on it you know sometimes those changes aren't going to go those constraints won't go away and so how can we maintain what makes fuse such a valuable experience even if those are some of the constraints that are that are being imposed on it so that's you know I think that's and that's it that's a big question that you know that a question is not going to go away so that's kind of what I mean well and just a elaborate a bit i mean one of the things that I think is especially useful in those conversations is to take the work that the evaluation team has done and suggested and offer it to our partners and saying you know here are some of the most thoughtful people in evaluating connected learning and thinking deeply about youth engagement in these contexts and here are some of the ways that that they've been able to identify important impacts beyond just counting the easiest stuffed account which is you know how many times that a kid do a challenge or something which we know is very syntactic and not at all you know relevant so so in fact we sort of see a pipeline if you will of ideas and practices that we're working out in the evaluation feeding back into the conversation with our partners in a way that I think is often kind of left out you know evaluation so it's tacked on at the end typically and we see it being a very kind of iterative dbr kind of you know an idea and so I just the other answer i would say bill is you know one of the things that's always frustrated me both in developing materials for formal as well as informal context is you know this this kind of holy grail that the NSF and other funders put out there which is you know we want to attract more kids get more kids interested in stem uh you know academic Korea you know pursuits majors in college or in careers but I've often found that the mechanism these sort of a one-off survey saying well what are you interested in now like oh yeah I definitely want to be a neurosurgeon you know I've always found you know those needing to be taken with some pretty hefty amounts of salt because you know kids change their minds especially middle schoolers and you know one of the things that I'm really excited about through this evaluation process is is trying to come up with a more robust way of documenting kids interest development over time not just something that happens on the answer or survey and then you know it's gone but something that we can really point to and say you know not only did they kind of say they were interested in it but we have other data that really reinforces this idea that they've chosen to spend time working on X Y or Z or exploring these other interests and so I think that that's you know that's going to be an important contribution to the field in general beyond the sort of you know smile sheets of you know interest surveys that's great so we're to that time where I want to offer opportunity to the people who are my gas theatre to present a challenge to the people who are out there that to actually help you with an evaluation problem that you might have so I'm wondering if you guys might present something it could be more than one thing about that you could use help with and implementing the evaluation plan that you have for few studios well what one thing that we're very interested in I think we're only at the very beginning of understanding this is trying to expand our data into the ways that participants physically move around the space and collaborate with each other you know we've talked about that the graphic data and we can pick up some of it from that you talked about the web data but we're really in after school environment or an informal environment like fuse where kids are physically moving around and that a movement through different learning Arrangements is really materially related to the kinds of activities that we're trying to understand I would really love it if we could get to a place where that's data stream of participants actual physical configurations in the space movement around the space could be automated or semi-automated in a way that could be folded in and used as another kind of threat of data sounds exciting Maggie or Daniella you want to add a challenge to the people participating online sure so something that comes to mind for me is we've been experimenting with lots of ways to visually represent the the the kinds of activity that that youth have at fuse and we have some pretty cool maps that we've already developed but i think that there's probably more that we could do so so we would be interesting to see what folks can suggest in terms of how to represent very fluid very dynamic types of activity patterns from you know from one challenge to the next over no long periods of time that's something i think that would you know would be pretty neat to see local researchers and for the facilitators mm-hmm and for program development even so we use a lot of the data we've talked about kind of research side but we do it for we use the the web data for internal evaluation as well so to make sure that our challenges are are meeting our own expectations in terms of engagement so there's there's lots of benefit there and lastly I would just add potentially an ethical way to incentivize youth taking the surveys more than once great because repeated surveys are very important for measuring change over time as we all you know these are great challenges and I want to invite everyone this week to respond to the challenges that have been posed by our panelists today that discussion will be happening in the forum for the course the URL for that is forum dot DML DML hub dotnet that is forum DML hub net and you can also continue the conversation from today using the twitter hashtag CL eval as much as possible though I'd like you to try something that I think will make your contributions really helpful to the panelists as you make them and that is when you go to a post see if there's an answer that you can build on or challenge or extend rather than starting a new topic or thread try to add to an existing one so that we can get a conversation going online and build out a suggestion that the panelist can use our next webinar will be februari 22nd at ten a.m. pacific time and then we'll be joined by rafi santo dixie chang and some participants from hive in new york and you'll hear all about what hive is then and they'll join me in our topic will be developing measures for evaluation that are tailored to programs in the world of connected learning will include a focus on measures for both connected learning and connected teaching so please join me in two weeks for this exciting session and please join me in thanking chemie and Maggie and Daniella for a wonderful session today and that's it for this the third session of our course on connected learning evaluation