Rewarding reviewers – sense or sensibility? A Wiley study explained

Reviewer motivation

A considerable amount of researcher time is spent reviewing journal articles. Rubriq (2013) calculated that approximately 30.5 million researcher hours was spent on reviewing. Taking just the top 12 publishers alone, it is estimated that at least 22 million researcher hours was spent reviewing papers in 2013 (Table 1).

What are the motivations for spending such a large amount of time reviewing? The results from our survey confirm the findings from other industry reports that reviewers choose to review because it allows them to actively participate in the research community and they feel it is important to reciprocate the peer review that they themselves receive.

However, there are some underlying differences by career stage. Reviewing as a path to improving one's writing skills, along with reputation building and career progression, is ranked higher by those who have been reviewing for fewer years, and therefore more likely to be in their early career (Table 2).

The top factor influencing the decision to accept a specific review invitation is the prestige and reputation of the journal, holding true across all age groups (Table 3). Also, both early career researchers and established researchers consider the prestige and reputation of the journal as the most influential factor on the time spent on that review, and their commitment to meeting review deadlines. The personal relationship and opportunity to network with the editor is ranked second for both groups.

This suggests that while reviewers choose to review in order to give back to the community, there is also more perceived benefit in interacting with the community of a top-ranking journal than a low-ranking one. Clearly, there are some reputation management and career progression factors involved here.

Who is bearing the reviewing burden?

Our survey indicates that 49% of reviewers currently review for five or more journals. Experienced reviewers, those with more than 5 years of reviewing experience, shoulder even more of the burden with 61% reviewing for five or more journals. There is some ambiguity in the responses here because of the phrasing of the question (‘How many journals do you currently review for?’). As the word ‘currently’ was not defined, reviewers may have answered according to either how many journals they are currently completing a review for or the journals on whose reviewer lists they currently sit. Either way, there is sufficient indication that at least half of respondents are undertaking, or could be asked at any time to undertake, a considerable volume of reviews.

In the survey, respondents from emerging and high-growth markets indicate that they are currently acting as reviewers for approximately the same number of journals as authors in mature markets (Table 4). In other words, they are ‘on the books’ for the same number (or more, in the case of China) journals as their US and Western European counterparts.

However, comparing the geographical distribution of reviewers undertaking a review, versus authors publishing articles in 2014, gives a different insight into reviewer workload. We looked at the top 10 countries (in terms of number of contributing authors) for Wiley health and life science journals using ScholarOne. This revealed that US researchers are reviewing 33% and 34% of the papers; in contrast, they are publishing 22% and 24% of papers. Chinese researchers, on the other hand, are publishing twice as many papers than they are reviewing (Figs 1 and 2). It has been suggested that a fair ‘reviewer commons’ is one in which researchers review the same number of papers as they themselves receive review for (Coin, 2015).

As editors frequently invite past authors to be reviewers, it is assumed that as the number of authors grows, so should the number of reviewers. This apparent regional imbalance could be one explanation for the increasing difficulty in finding reviewers, and also the burden of a growing number of articles without a commensurate growth in reviewers.

Anecdotally, some editorial offices observe that Chinese researchers have one of the highest review invitation acceptance rates. As with researchers across the full pool, Chinese researchers rate being an active participant in the research community as the most influential factor in choosing to become a reviewer with mean rating of 4.3. Perhaps unsurprisingly, improving their own writing skills (3.8) and gaining professional recognition (3.8) receive higher mean rankings from Chinese researchers than the overall mean across all regions.

This apparent uneven geographical spread raises some interesting questions. It is possible that more emphasis is being placed on the importance of authoring over reviewing in some high-growth markets such as China and within emerging market institutions. But, when asked, high-growth and emerging market reviewers demonstrate an appreciation of the value of reviewing activity, for the benefit both of the research community and personal development.

So, why is there an apparent imbalance in reviewing versus authoring effort? Is there an issue here around editorial trust and confidence in the reviewing standards of emerging and high-growth market researchers? Is it more difficult for editors from international journals to find referees from, for example, China, because they do not know where to look? Or are there generally fewer experienced/skilled reviewers in these areas? Could training support help attract more reviewers from these areas to the pool, provide evidence of reviewing skill, and possibly help alleviate the pressure elsewhere?

What type of training do reviewers receive?

It has frequently been asserted that there is a lack of guidance about how to perform a good review, and reviewers are expected to ‘learn on the job’ (Sense About Science, 2009). This is borne out in the responses to our survey about the types of training that reviewers have previously received.

The most common type of peer review training received by all respondents to date has come either in the form of guidelines, for example a journal's instructions for reviewers (32%), or Committee on Publication Ethics (COPE)'s ethical guidelines (18%), or informally as advice from supervisors (19%) or colleagues (16%). This also holds true across all regions. Fewer survey respondents indicate that they have experienced more formal training, participated in a journal's reviewer mentoring programme (4%), attended a workshop/seminar on the topic (4%), or watched a video or webinar (3% and 2%) (Table 5).

When asked what type of training reviewers would find most useful, journal and publisher guidelines continue to be ranked highest overall as useful resources for reviewers. However, there are some differences in the rankings between early and late career respondents. Early career respondents rate guidance and mentoring as important, while late career respondents rank general ethics guidelines for peer reviewers as more important.

What training needs do reviewers have?

How confident are reviewers in their reviewing skills? On average, reviewers self-assess their skills at 3.7 out of a possible highest rating of 5. Not surprisingly, those with less reviewing experience rate themselves less skilled than those with more experience, and those who have reviewed fewer than 10 papers have an average mean score of 3.4 compared with 4.2 for those reviewing over 100 papers to date. Reviewers in Asia have the lowest average mean score of 3.4 compared with US reviewers who have the highest average of 4.0, although this is perhaps more indicative of cultural differences and lower baseline self-assessment levels than an accurate reflection of reviewing skill.

Despite expressing relatively high levels of confidence, 77% of reviewers express an interest in receiving further training. As we would expect, demand is particularly strong (89%) among respondents with 5 or less years of reviewing experience. However, established career researchers also express an interest in training (75% of those with 6–10 years of reviewing experience and 64% of those with 11–15 years of experience) (Table 6). Notably, it appears to be the fundamentals of reviewing such as constructing a review report and providing constructive, useful feedback that consistently elicit the highest interest across all experience levels (Table 7).

Responses show some variation between disciplines. There is higher demand for training in how to review a qualitative research article in the social sciences and humanities, and greater demand for training in performing a statistical review, reviewing a systematic literature review, reviewing data, and handling re-reviews in the health and life sciences. In addition, there is specific interest in how to review a clinical paper from health science respondents.

There are also some regional differences. Asian reviewers express much higher demand for an introduction to becoming a peer reviewer, working with editors, and reviewing a qualitative research paper than Western counterparts.

Recognition for reviewing

Reviewers strongly believe that reviewing is inadequately acknowledged at present and should carry more weight in their institutions' evaluation process. Moreover, respondents say they would spend more time reviewing if their institution recognised this task (Table 8). Some work was undertaken in this area in 2012 when Sense About Science's Voice of Young Science Network lobbied for the Higher Education Funding Council for England to recognise peer reviewing in the UK Research Excellence Framework (Sense About Science, 2012), and in 2015, a group of Australian Wiley editors, led by Associate Professor Martha Macintyre, published an open letter to the Australian Research Council (Meadows, 2015), requesting for greater acknowledgement of the importance of editing and assessing research within the Excellence in Research in Australia assessment system.

But putting institutional recognition aside, what type of recognition do reviewers want from their journals and publishers?

Survey respondents were shown a list of reward and recognition initiatives and asked to select the ones that would make them more likely to accept an invitation to review. Individual initiatives were grouped into categories – acknowledgements, accreditations, rewards, performance-based rewards, and feedback. Three of the top six most selected individual initiatives are related to receiving feedback from the journal – on the quality of their review, learning about the decision outcome, and seeing other reviewer comments (Table 9).

Next to this, the second most valued category of reward is acknowledgement, whether in the printed journal, on the journal website, or a personal note from the editor. Respondents are definitely more interested in receiving feedback and editor/journal ‘thank you's’ or recognition for their reviewing efforts, than cash or in-kind payments, although receiving access to journal content also featured highly (Table 9).

In many ways, this is consistent with what we have learned about reviewer motivation. As covered earlier, reviewers are motivated by their desire to actively participate in their research community - they want to know that their contribution has been well received and was worth the precious time they spent.

The four most preferred reward and recognition initiatives hold true across all markets. However, responses from reviewers in high-growth markets indicate that acknowledgements in the journal or on its website are less important than receiving access to papers they have reviewed, or a digital ‘Top reviewer’ badge that could be displayed on personal and social media websites. Reviewers in mature markets show a higher preference for discounts or waivers on Open Access fees.

A Need To Increase The Reviewer Pool

In order to reduce reviewer workload issues, there is a need to increase the pool by attracting early career researchers and new markets, including reviewers from high-growth and emerging regions. However, the findings with regard to the apparent uneven geographical spread of reviewing burden suggest that there is also a need to make sure that the work is spread out evenly.

There may well be process improvements that publishers could make to better identify and record possible reviewers. Increasing use of customer insight tools, either those developed in-house or by vendors such as DataSalon (http://www.datasalon.com) or SalesForce (www.salesforce.com), may offer opportunities to more easily identify potential reviewers.

Formal recognition from research assessment bodies of review activity as a measurable research output [facilitated by a taxonomy of contributor roles, such as project CreDit (http://casrai.org/CRediT), and unique identifiers for reviewers, i.e. ORCID (http://orcid.org/)] could help alleviate some of the time allocation issues, as reviewing would be seen as part of researchers' activity rather than an additional commitment.

With specific reference to Chinese reviewers, further research is needed to assess the extent to which the imbalance is due to any or all of the following factors: a lack of recognition for reviewing activity, which may be causing less Chinese researchers to undertake reviews; a skills deficit, which may mean that Chinese reviewers are less confident at reviewing for international journals; a possible reluctance of international journal editors to use Chinese reviewers (due to either real or perceived skills gap); and/or difficulties in identifying potential reviewers from China.

Our findings suggested that journal rank plays an important role in the decision of researchers to accept a review invitation. This could imply a tiered reviewer market, where researchers are more willing to review for higher impact journals. If this is indeed the case, lower-ranking journals may need to work harder or employ different tactics to attract and motivate reviewers, perhaps in the areas of reviewer mentoring or reward. However, further analysis is required to fully determine whether lower-impact journals find it more difficult than high-impact journals to recruit the requisite number of reviewers or encourage reviewers to deliver reviews on time and to the highest standard.

Core competencies – a possible solution

Publishers need a consistent answer to the question, what makes a good reviewer? Studies have tried to quantify the characteristics of a good reviewer (experience, proven review frequency, etc.), but these studies are primarily based on quantitative factors (Black, van Rooyen, Godlee, Smith, & Evans, 1998). Alternatives, such as looking at attributes based on the profile of current reviewers, could just serve to reinforce existing distribution of reviewing effort.

One possible solution is the concept of core competencies as advocated by Moher (2014) and others (Glasziou et al., 2014). If we want to train reviewers effectively, and also measure their performance, perhaps there is a need to establish an industry-wide set of minimum core reviewer competencies. A set of universally agreed reviewer competencies, with some variation at subject level, could provide the basis for both a training framework and ongoing measurement and evaluation of reviewer quality.

Earlier in this article, it was asked whether there is a lack of trust in the reviewing ability of emerging and high-growth market researchers. A training and recognition mechanism based on core competencies could help alleviate this issue. There is an opportunity for centrally funded reviewer training programmes, delivered by publishers who have the expertise and the content, tailored according to regional needs and designed to deliver learning outcomes based on these competencies. In 2014 alone, over 30 editorials in Wiley journals offered guidance on peer reviewing. These were some of the most highly viewed articles in the year, indicating a strong researcher interest in more information and guidance on reviewing.

Finally, this same competency framework could also be used to provide consistent and meaningful feedback on review activity to more established reviewers. The recently launched Think. Check.Submit campaign (thinkchecksubmit.org) is a good example of what can be achieved by a coalition across the field of scholarly communications. Further collaboration could greatly facilitate progress on this issue also.