The Art and Science of Effective Paper Reviewing

Authors who submit papers to professional journals believe that their work is worthy of publication, but many journals report acceptance rates of less than 50%. Authors who receive negative decisions often feel that the review process was not fair to them. Editors try to ensure that the reviewing of a paper is comprehensive. Also, editors are continually trying to improve the review process. Information about the review process and reviewers’ perspectives may be of value to all authors in preparing papers, which will help them be more successful in having their papers published in journals of their choice. Writing a paper that will be well received by editors and reviewers is partially an art, as decisions are not solely based on the technical content.

The goal of this editorial is to make potential authors aware of the multitude of factors that influence decisions as to the suitability of a paper for publication. Since the assessments by reviewers are the primary decision tools, the focus of this editorial will be from the perspective of the review process.

It may be worthwhile to briefly summarize the review process. The review process can be viewed in terms of three stakeholders: the authors, the editors, and the reviewers. Each stakeholder has rights and responsibilities. When all of the stakeholders efficiently meet their responsibilities, the review process works well and the journal contains quality papers. When an author has submitted a paper to this journal for review and the file has been cleared by the editorial staff, the editor-in-chief (EIC) checks to ensure that the paper is technically appropriate and readable and, if so, assigns the paper to a section editor (SE), who has the responsibility of identifying an appropriate associate editor (AE). Having three levels of editorial review may seem redundant, but helps to distribute the work load and, more importantly, ensures a broader assessment of papers. The AE then identifies typically three reviewers who are knowledgeable about the topic of the paper. Reviewers may be selected from an electronic database through the use of keywords, or the AE may select experts because of their reputation in the field. Ideally, persons who are invited to be a reviewer will respond to the request in a timely manner whether they will accept or decline the review. Once reviewers electronically submit their assessments of the paper into the Editorial Manager system, the AE evaluates the reviews to ensure that the comments are reasonable and unbiased. The AE may provide additional comments that are intended to help the authors improve the paper. The folder is then transferred to the SE, who reviews the file, makes additional assessment notes, and forwards it to the EIC who renders a decision and may provide additional comments. This multilevel editorial review is intended to ensure that submissions receive a fair assessment with comments on changes that can lead to a paper of high technical quality. Each editor has the responsibility to ensure that the reviews are conducted in a timely manner.

Since this editorial focuses on paper review, the spotlight will be on the responsibilities of the reviewers. For optimal results, reviews should be timely and comprehensive. First, it is important to reflect on the fact that reviewers are not reimbursed for their efforts, as they volunteer their time to provide paper reviews. Second, if the reviewer believes that reviewing the paper would represent a conflict of interest, he/she should contact the AE and explain the issues involved. Third, confidentiality is important; therefore, the reviewer should not share the paper with others without contacting the AE and explaining the reason that input from another party would be beneficial to the review. Fourth, if the reviewer suspects plagiarism, fraud, dual publication, or other issues with ethical implications, the AE should be contacted for advice. The primary responsibility of a reviewer is to evaluate the accuracy of the technical content of the paper and provide an opinion on the value of the work to the profession. This includes assessment of the relevance and timeliness of the material. The paper should use state-of-the-art methods. It is important that a paper be based on a novel idea. Authors should reference previous works that reflect current knowledge, and then clearly emphasize how the paper advances the state of the art beyond current work. The authors’ failure to show advancements in knowledge is a common reason for a reviewer recommending that a paper be declined. The reviewers determine whether or not the content of the paper is based on outdated methods or provides a novel approach to a problem; this as an important decision criterion. Also, a paper should be balanced and fair when making comparisons to existing methods and should provide information of interest to a broad readership; reviewers should assess the extent to which this issue is addressed by the authors.

When assessing the quality of a paper, it is worth repeating that reviewers tend to focus on novelty, i.e., the originality of the work, as this is the primary indicator of an advancement of the state of the art. A significantly new model, method of analysis, or analytical or numerical approach to solving a problem is usually considered a novel advancement; however, the improvement of an existing model or method may be innovative and reflect sufficient novelty to be accepted. When authors fail to emphasize the novelty of the research, the value of the paper may not be recognized by the reviewers. By emphasizing the specific nature of the innovation of the research, the authors can impress the reviewers and editors as to the value and uniqueness of the work. The research goals and objectives should be strong statements that indicate how completion of the research represents an original approach to solving the problem; however, the statement of the research goal as well as the objectives should not be limited to statements that support the novelty of the work. The uniqueness of the approach may also be included as an objective. When discussing the state of the art of the paper’s topic, the novelty of the work should be discussed in a positive way that does not present existing methods in a negative light.

Secondary aspects of a reviewer’s assessment center on two important parts of a paper: the Introduction and Conclusions sections. The introduction should include a clear statement of the research goal and objectives. Often, authors insert a summary of the paper instead of statements of the research objectives, thus inviting criticisms from reviewers. As part of the review, a reviewer should evaluate the conclusions of the paper to ensure that they are more than just a summary. Brief summaries that are little more than the material presented in the abstract are of little value in achieving a quality paper. Instead, the broad implications of the research results should be addressed in the Conclusions section.

The third level of review criteria should focus on the methodology and discussion of results through proper use of figures, tables, notation, and paper length, as these can convey important knowledge and influence the readability of a paper. Do the figures and tables adequately support conclusions used to substantiate that the research objectives have been met? Reviewers should address this question. The format and legibility of the figures and tables are important as they communicate important knowledge to a reader. The font size, thickness of plot lines, and color gradations should be considerations when composing a figure. The titles to both should be descriptive, not simple statements. Figure and table titles can be used to define variables, notations, and units. A figure or table should be understood by a reader without reference to the manuscript. The axes of a figure should be clearly identified with the units shown. If a figure contains multiple symbols, they should be identified in the title or as part of a legend. Poorly composed figures can be ineffective in transferring knowledge. For example, a graph of measured and predicted daily discharge rates versus time that covers a period of 10 years will be so crowded that the reader will be unable to assess the ability of a model to accurately estimate the measured values. Reviewers should identify figures or tables that are not necessary or could be condensed. If the notation in the paper is extensive, a reviewer may recommend that an appendix of notation be provided. Paper length can influence the amount of knowledge gained by a reader. An extremely long paper can result from excessive detail or redundancy in the material presented. Also, journals have page limitations because publication costs are significant. Reviewers consider the length of the paper in their assessment, which ensures that the knowledge content is worthy of the page count; therefore, authors need to consider length when composing a paper.

Identifying errors in grammar and typos should be a minor element of a review unless they prevent the technical content of the paper from being understood. Authors who struggle with grammar should seek an independent review of the paper prior to submission to ensure that the content of the paper can be understood. Failure to do so increases the likelihood that the paper will receive negative reviews. A reviewer may conclude that poor grammar may be masking the quality of the presented work and that an external review of the grammar should be sought prior to submitting a revised draft. After the final decision is made, papers accepted for publication undergo a review for grammar.

Just as the organization of material in a paper can be a decisive factor for a reviewer, the presentation of the review is important to the authors. The format used by a reviewer can help the author’s organize replies to the reviewers’ comments. Well-organized reviews can also assist the editors in assessing the review comments to ensure that they are fair. Reviewers should number each separate comment. The authors can use the same numbering system in presenting replies to each comment. When the reviewer’s comments are extensive and address different aspects of the presentation, headings that separate a review into sections can be helpful.

Reviewers can assist the editors in making a decision by distinguishing between required and recommended changes. All comments to the author should be very specific, as it is difficult for authors to make changes when the comments are vague. A reviewer should review his or her comments to ensure that they will adequately communicate the concerns to the authors. In turn, the authors should clearly and fully support their judgments, so that the editors and reviewers are able to fully understand the reasoning behind the statements. Required changes are those that the reviewer believes would be necessary to make the paper acceptable for publication. If the required changes are significant, then it may be best to decline the paper and recommend that it be resubmitted after extensive changes are made. Recommended changes are those that the reviewer believes will improve the quality of the paper, but failure to make these changes would not necessarily keep the paper from being acceptable for publication. Authors and reviewers should consult ASCE’s definitions for recommendations available in the Editorial Manager (http://ascelibrary.org/doi/pdfplus/10.1061/9780784479018). A reviewer should ensure that each recommendation is commensurate with the comments that are provided in the review.

Even high-quality papers may require changes and a re-review of the revised manuscript. If the initial review was comprehensive, then the re-review should be completed quickly. The editor and reviewer will only need to check to ensure that responses have been provided for all comments and that the reviewers’ comments have been adequately addressed. Generally, each of the required changes should result in a change to the paper. Recommended changes may also result in changes to the paper. If the author elects to not make a change to the paper based on a recommended change, then it is important for authors to provide a full explanation. The replies to the reviewers should be conciliatory, rather than combative, in tone. All stakeholders, i.e., the author, editors, and reviewers, want to produce quality papers that will simultaneously enhance the reputation of the journal and move forward the state of the art.

Unless extenuating circumstances arise, a reviewer should expect to be available to re-review the revised manuscript in a timely manner to ensure that the review comments were adequately addressed. If a reviewer cannot provide a reassessment of the revised paper, especially when the suggested changes were substantial, then it is necessary for the AE to find a replacement reviewer; this is not desirable as it generally extends the review time needed to process the paper.

If all stakeholders understand their responsibilities, then published papers will be of high quality, which enhances the quality of the journal.