Reviewer’s Checklists for Evaluating Scientific Manuscripts

Peer review is a critical step in the process of publication of a scientific manuscript. The aim of reviewing is to evaluate the quality of a manuscript on various aspects and to make a recommendation to the editor for publication in the concerned journal. A good review considerably increases the quality of a manuscript and is beneficial for the reviewer and the author in creating future manuscripts.

While searching for reviewers, the editors tend to pick those who have accepted most invites as well as those who have completed most reviews, since these data are available in the database. It is difficult to predict a high quality reviewer based on the qualifications, training and experience [1]. Being invited as a reviewer is a recognition of expertise in that particular field.

It has been found that spending more than three hours in reviewing a manuscript did not increase the quality of a review but the quality improved with increasing time spent up to 3 h [2]. Younger reviewers up to the age of 60 years, training in statistics or epidemiology and current research investigator were significantly associated with good review quality according to assessments by editors [2].

The role of a reviewer starts when he/she receives an email from the editor inviting to review a manuscript. The information provided to them would be the title of the manuscript and the abstract. The reviewers have to make a decision to review or not based on this information depending on whether it falls within the remits of their area of specialization and expertise. It is advisable to accept or decline the invitation as soon as possible as it will give an opportunity to refer to other reviewers early and saves time for the editors and the authors. Since the publication by Atul Gawande on checklists in surgery and adopted by WHO in preoperative checks in operation theatres, their use has been established and growing in medicine [3, 4]. While there are exhaustive lists for assessing each type of article, we aim to provide useful basic checklists for reviewers while evaluating manuscripts and formulating their reviews.

What Should the Reviewers Equip Themselves with Before Accepting a Manuscript?

  1. 1. Scope of the journal
  2. 2. Basic working knowledge of the subject being reviewed
  3. 3. Good English and grammar
  4. 4. Knowledge of what information goes into each section of a manuscript. (Introduction, Materials and methods, Results, Discussion, Conclusions and References)
  5. 5. Awareness of author instructions with reference to
    1. a. Tables (should be less than 7 combined with figures for IJO)
    2. b. Figures
    3. c. Word count
    4. d. Format of references cited in text as well as bibliography

    During Review

    Perform a basic scan of the above aspects of the manuscript. Tables 1 and 2 will aid in the basic review process of scientific manuscripts.

    Table 1 Checklist during the initial assessment by the Editor and the Associate Editor (All contents under acceptable and satisfactory are in green colour shading and those under unacceptable and unsatisfactory are in light red colour shading)

    Table 2 Checklist for evaluating manuscripts for the reviewers (All contents under acceptable and satisfactory are in green colour shading and those under unacceptable and unsatisfactory are in light red colour shading)

    The colour coding of the tables would give an idea regarding the decision to accept or reject a manuscript. If the manuscript predominantly scores green shades (from Tables 1 and 2) on most issues mentioned, then the manuscript is probably good to be accepted provided major or minor revisions are not required. This objective tool will also help in rejecting a manuscript that is unsuitable when the manuscript scores red on most issues. Though this is likely to help decision-making, not every manuscript can be judged in green and red shades, some are likely to have grey areas, and herein the reviewer will be required to exercise discretion. Similarly, a checklist to review statistics of a paper may be used. How to present statistics in a paper has been given by previous publications [5,6,7].

    The following is a summary of the most important Dos and Don’ts during reviewing.

    Dos

    Don’ts

    Reviewing a Revised Manuscript

    This is usually quick since the manuscript has been reviewed once before and only the responses by the authors need to be checked along with corrections made. Hence, the time given for reviewing a revised manuscript is usually a week by most journals.

    1. 1. Read all the inputs from other reviewers and the responses given by the authors to the reviewers.
    2. 2. Have the authors addressed all the issues raised by reviewers satisfactorily?
    3. 3. If yes, countercheck the changes made by the authors to the manuscript
    4. 4. If no, politely give a reminder. The author may have forgotten to resolve some issues.

    Editors’ Role in Reviewing

    1. 1. The editor picks the reviewers who can do a good review of the manuscript. He/she favours those who respond fast, provides good quality reviews on time and is reliable. The above information on reviewers will be available in the database. Experienced reviewers are usually preferred for obvious reasons. If you are early in your career as reviewer, and provide a bad review, you may not be favourably considered for another review. From the editors’ point of view, it is valuable time wasted by the reviewer. This delays the editing process. So new reviewers should take this learning opportunity well, take time to meticulously analyse and then provide the review. A good start is to practice critiquing manuscripts in journal club meetings before taking up reviewing.
    2. 2. Editors review the manuscript they are assigned and are guided by the inputs from reviewers to make a decision on publishing.
    3. 3. They peruse the comments from reviewers and combined with their own assessment, make a plan and take a decision on the document regarding its publishability.
    4. 4. They act, as a link between the authors and the reviewers in improving the quality of the manuscript since reviewing is a blinded process.
    5. 5. Some of the comments by reviewers may be edited or deleted before they are sent to the authors if the editors find the comments inappropriate [8].

    Tips for Book Review

    Book review should be undertaken by an expert in the particular field e.g. a foot and ankle surgeon writing a review of a book pertaining to upper limb would not be considered appropriate. These are our recommendations for writing a review on a book.

    Controversy Surrounding Pre-print Servers

    Authors and reviewers need to be cautious while submitting to preprint services such as MedRxiv because manuscripts submitted to these preprint repositories are likely to be picked up by the plagiarism detection software used by reputed journals and likely to show higher similarity. In addition, manuscripts once submitted to preprint servers could not be withdrawn. Plagiarised and unscientific manuscripts are reported to be excluded from these preprint services and the biggest drawback as acknowledged by these preprint servers is the lack of peer review. Since these manuscripts are cited on Google scholar and Crossref, there is a potential for dissemination of non-peer reviewed work to enter public domain and erroneously influence decision-making.

    References

    1. Callaham, M. L., & Tercier, J. (2007). The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Medicine,4(1), e40. ArticleGoogle Scholar
    2. Black, N., van Rooyen, S., Godlee, F., Smith, R., & Evans, S. (1998). What makes a good reviewer and a good review for a general medical journal? JAMA,280(3), 231–233. ArticleCASGoogle Scholar
    3. Weiser, T. G., Haynes, A. B., Lashoher, A., Dziekan, G., Boorman, D. J., Berry, W. R., & Gawande, A. A. (2010). Perspectives in quality: designing the WHO Surgical Safety Checklist. International Journal for Quality in Health Care,22(5), 365–370. ArticleGoogle Scholar
    4. Laws, E. R., Wong, J. M., Smith, T. R., de Los, R. K., Aglio, L. S., Thorne, A. J., Cote, D. J., Esposito, F., Cappabianca, P., & Gawande, A. (2016). A checklist for endonasal transsphenoidal anterior skull base surgery. Journal of Neurosurgery.,124(6), 1634–1639. ArticleGoogle Scholar
    5. Squires, B. P. (1990). Statistics in biomedical manuscripts: what editors want from authors and peer reviewers. Canadian Medical Association journal journal de l’Associationmedicalecanadienne,142(3), 213–214. CASGoogle Scholar
    6. Curran-Everett, D., & Benos, D. J. (2004). Guidelines for reporting statistics in journals published by the American Physiological Society. Physiological Genomics,18(3), 249–251. ArticleGoogle Scholar
    7. Curran-Everett, D., & Benos, D. J. (2007). Guidelines for reporting statistics in journals published by the American Physiological Society: The sequel. Advances in Physiology Education,31(4), 295–298. ArticleGoogle Scholar
    8. Hamilton, D. G., Fraser, H., Hoekstra, R., & Fidler, F. (2020). Journal policies and editors’ opinions on peer review. eLife,9, e62529. ArticleGoogle Scholar