Electronically Signing Peer Review Report Made Easy

Electronically signing peer review report

next up we have Elizabeth Elizabeth Seaver from P Lois Mose P Louis excuse me hi I'm Elizabeth Seaver I'm a researcher at the nonprofit open access publisher Public Library of Science or PLoS or P loss or P we call it bloss and my co-author Helen Atkins and I are both employees applause the study I'll be presenting includes data that we collected at PLoS so consider that our conflict of interest so first to set up the topic of sign to peer review we should define it so we've talked about how there are lots of different definitions of open peer review for signed peer review specifically we're talking about reviews where the reviewer reveals their name to the author in a typical closed peer review session and as other speakers have alluded to the practices around this vary widely by field I'm from the social sciences where double-blind peer review is is the most common in the life sciences single blind review is very common and you know a lot of the pioneering work in ensign reviews comes from some medical journals like the like the BMJ so there are a number of journal editors who express some hesitation about moving towards a sign of you model one of those kinds of concerns can be that signed reviews might be lower quality because people feel they can be less honest but there are there have been many studies that support the value of signed reviews where they found that they are more constructive as rated by authors and editors that they that they maybe will take a little bit longer to to receive or there might be a little bit of a lower review acceptance rate but that overall they can improve the experience so for Plus journals specifically a Plus has a suite of 7 journals and for all the journals plus encourages reviewers to reveal their name to the author but it is not required and the prompt is very similar on all the journals where it says if you wish to reveal your name to the author you know put it in this box one of our journals plus medicine also includes a yes/no checkbox for whether you'd like to sign your review so in the data I'm going to be presenting I'm going to be focusing on three plus journals specifically so plus one is our large quote/unquote mega Journal multidisciplinary very very diverse research community four plus medicine that's one of our flagship journals most of the editorial staff is in-house and then plus computational biology which is one of our community journals is the computational biology a community which also includes a bioinformatics and so something that might be helpful to also know about this particular research group is that they tend to be you know at the forefront of a lot of open science endeavors they're some of the early adopters of the archive you know largely because there's a big overlap with a computer science so they're posting they were posting preprints you know earlier than other areas of biology and also they contribute a lot to the open source movement and use a lot of open source tools and of course plus journals are open access so there's a lot of openness going on here so with this data we considered two types of measures so one is a behavioral measure which is just how often our reviewers of these plus journals signing their reviews and also what are their attitudes about signing what do reviewers and authors think they're prefer about about signing reviews so first let's look at some of the signing data so this is the percentage of reviews signed by year from 2013 to 2016 at these three journals and you'll notice that for both plus computational biology and plus one the rate is pretty consistently low between about five and ten percent and that holds true across the years and plus medicine on the other hand has a much higher signing rate of varying between about thirty and forty percent with a slight dip in the most recent year but you'll notice that there is there is a very large difference and that plus medicine seems to have a lot more review signed so then we want it to dig in a little bit and see is there a difference in the kinds of articles that are getting signed reviews or not and so what we did is for a subset of the articles from 2015 and 2016 we looked at the final editorial outcome for for this set of reviews that is whether the reviews were published or rejected it appears that the rejected key has disappeared out the screen but - you might be able to guess that the the brown bars in this case mean reviews for articles that were eventually rejected and so you'll notice that the pattern between the three journals is consistent again the signing rate for class medicine is higher but also across all three journals there's a consistent difference between reviews for articles that were eventually published or rejected in that for published articles there's a slightly higher percentage of reviews that are signed this might offer some credence to a one theory about why reviewers might be hesitant to sign if they're worried about a retribution but but it's hard to know exactly what the cause is so that's the behavioral date that we have now let's move on to the survey data where we asked both reviewers and authors about their views on on signed review and starting with reviewers so this data is not linked you know one to one with the previous data this is a separate fully anonymized data set we asked reviewers in 2015 to 2016 how often do they sign reviews and we excluded first time reviewers from this because you know they haven't been reviewing enough to have a trend in one direction or another and so we asked if they signed usually sometimes or never is the is the Graybar and so you'll notice that for fourplus medicine again the rate of reviewers who usually are sometimes sign is higher than for the other journals but for plus computational biology the self-report of how often they sign seems to be you know also a lot higher for end 4+1 but seems to be a lot higher than the actual number of reviews that are that are signed showing a sort of mismatch potentially for those two journals in self-reported you know rate of review signing and actual review signing then we also wanted to get a sense of what authors actually prefer do they do they like signed reviews or not so in this in this slide here teal means that they usually sign and watch what happens to the teal and the next one so this is the proportion of authors who say that they prefer to receive either a signed review until an unsigned review in grey or neither in in purple and so apologies for the small sample sizes for the for the last two journals in this case but all of this supports you know previous research in that authors tend to be in favor of a signed review and they're very least open to the idea you'll notice that those who declare an explicit preference for unsigned review are you know pretty a pretty small group and it mean it makes sense like what what do authors get from sign review they get more information about who's you know evaluating their work it's there's there's not really much of a direct downside to authors for that and then we also wondered you know how much of this preference might be due to experience with sign review and their proportion so the proportion of authors who actually report that they have received sign reviews before is lower than those who say that they would potentially prefer that for for all the journals and for plus medicine is that fifty percent so even though authors might not have a huge amount of experience with this this is still something that they are very open to and again we included we excluded a first-time authors because they wouldn't have a lot of experience with whether they have or haven't received sign reviews so in addition to asking these multiple choice questions about about their experiences we also asked follow-up questions in more and more depth and there were several themes that emerged from their comments so for reviewers who prefer to sign reviews they listed some of those benefits as that they felt that it improved accountability and the constructiveness of their reviews for those who prefer not signing they've a mirror you know some of the concerns I mentioned that editors can have where they felt they could be more honest and were safe from retribution but there was also a good number of reviewers who well they said they haven't signed reviews it wasn't that they didn't have a preference they said well I've never been asked before and I'm not really sure of the benefits you know why would I sign so it's just that incentive structures maybe not laid out clearly and then for authors about when we asked why they preferred signed reviews they they mentioned being able to learn the reviewers area of expertise and a potential for more open communication as being especially desirable so if you're a journal and you want to encourage more review signing what might you be able to do about that one thing you can do is directly request a signature that's also that's a good start and you know since reviewers mentioned potentially not being aware of incentives provide incentives directly or at the very least describe what some of those benefits of signing might be and again if you remember the discrepancy between reviewers and authors and their preference for signed reviews reviewers are also authors so it might be helpful to encourage them to take the author's perspective of how they as authors might benefit from receiving a signed review as opposed to unsigned so there's a lot of room here for exciting future studies so some things that we'd like to look at would be identifying more factors that would be affecting review signing identifying other kinds of differences between signed and unsigned reviews experimenting potentially with different signing prompts because the user experience of actually going through the review you know I mentioned that we have a text box where they put their name but there's a lot of ways that you can manipulate that and there's a very cool differences between journals and how they actually the actual experience for signing reviews that could potentially be driving behavior and communicating you know implicit norms about signing in that particular community and with that journal and it would also be great to be able to compare the attitudes and practices directly where we could do a one-to-one mapping knowing rich reviewers signed or not and went whether they generally sign or not as in their self-report so I'm happy to take any questions thank you Scott Lamar from Baylor College of Medicine and journal surgical research thank you for that terrific presentation how many individuals that answer the survey as authors also answered the survey as reviewers assuming though there must be some overlap between those two groups and if you if you were able to look at that to what degree was there concordance or discordance between the attitudes in other words were there people who said in from an author perspective yes I'd like to have sign reviews but yet when they answered as a reviewer never sign reviews but that's a great question and unfortunately because the data was fully anonymized with the way we ran the survey we weren't able to you know cross tabulate that however in their follow-up comments both authors and reviewers pointed to you know the perspective of the other role like for example some authors who said they don't they didn't want to receive a signed review might say well if I were the reviewer I wouldn't want to sign my name and some reviewers you know in you know identifying sort of potentially more benevolent aspects of signing reviews or saying you know it can help out the author so there was because most authors and reviewers are both right and so there was acknowledgement of those different roles but we don't unfortunately know the overlap for our data Thanks you showed the relationship between whether somebody's signed to review it didn't sign the review with whether the manuscript was accepted or not accepted so that's at the manuscript level did you actually look at the individual review level whether it was a recommendation to accept or not accept or just a negative review or a positive review in general right so you did I I passed it there we go right so you're talking about whether is right so in this case on the other bars represent the reviews themselves and so they're compared for the eventual editorial outcome so they didn't know at the time there's a difference between the journals whether the reviewers actually make a recommendation to publish or not publish so I believe for A+ medicine they are plus one they do actually make a recommendation and I'm not sure that that's the case for plus medicine or Plus computational biology in any case we didn't actually look at the content of the reviews themselves to see if they are positive or negative because it is possible for some of the ones that were rejected the reviews where they were assigned could be positive or negative so we didn't actually do that kind of analysis Thanks this wouldn't be directly addressed in your survey but I just wondered if you had any sense because you will be landing up in situations where you have some reviewers who signing and some who not is that landing is that leading to any complications in the process because some reviewers are signing or not signing the identity of some they don't know is that affecting how they're addressing the reviews is that affecting how the editors are considering that I'm just wondering if that affects the decision in any way mmm that's it that's a great question not that I'm aware of although I'm not one of the people on the front lines of the you know moderating that sort of process I do have some some theories about that I'm one one future study I would like to do would be looking at the signing rate of the first review versus subsequent rounds of revision because one thing that I do hypothesize is that if reviewers see that somebody else has signed they might be more likely to sign a later one but future studies will have to look at that see everyone BMJ but i used to be the editor-in-chief of PLoS Biology so we operated this some people signed some people don't and actually what I think is really interesting is the author's ignored the fact that they knew who someone was and they always reply reveal one said this and reveal two said that it's like you do know who reviewer 3 is right their name so I think there's a sort of sense that they're all equal and some of them just happen to have included their name and I don't not data just wondered your sample sought your response rates vary from 3 to 25% and there would be a concern that there's response bias here and I wonder whether you could anticipate what if people who are more interested chose to respond do you think they would be more Pro signing a more anti signer or how much do results be biased yes a response bias is gonna be a problem with any survey so based on you know the length and passion level of the comments I would say that you know there wasn't a clear bias in the responses for signing or not signing the people had very strong views in both directions so my guess would be that just the level of personal belief or investment and how important signing reviews was or how important anonymity was would be will be driving responses as opposed to a particular view of one direction or another it's people get very passionate about this and Gregory and circulation and previously Medical Journal of Australia asking on behalf of early career researchers who may be invited to review due to the death of reviewers available or wanting to spend time would you allow co-signing where a junior researcher can be supported by a senior who maybe has checked their review prior to posting so I think you know co-signing is a very interesting and complicated topic because one of the things that happens in practice is that sometimes you know graduate students or postdocs will essentially ghost write reviews for their for their PI so then if you want to we don't need to get into the issue of getting credit for peer reviews as actual valid scholarly documents but I think that in general you know if there's if people are going to be signing reviews you'd want the actual authors to be signing you know if if the P I is participating in that way like having their name on there makes sense in that there and author of the review they don't further questions thank you so much thank you [Applause]