Reproducible Results Policy

To further improve the reproducibility of work published in the Journal of Water Resources Planning and Management, narrow the gap between research and practice, and promote reproducibility as a moral and ethical imperative in our practice of science and engineering, the editorial board now encourages authors to add a “Reproducible Results” section immediately after the “Data Availability Statement” section in their manuscript. To incentivize authors to make their results more reproducible, the Journal will publish technical papers and case studies with verified reproducible results open access free to the authors for the next year. In future years, the Journal will publish technical papers and case studies with verified reproducible results open access either free or for a reduced fee, as funds are available. The Journal will also recognize papers with reproducible results in a new special collection and offer two new annual reproducibility awards for authors and the people who assess the reproducibility of results.

Authors of any article type may add an optional “Reproducible Results” section to their manuscript if all research materials listed in the “Data Availability Statement” section are publicly available in a repository or online. Next, authors must follow best practices to improve result reproducibility (Rosenberg et al. 2020). One best practice is to have a colleague, student, or other person not affiliated with the study reproduce manuscript results. In the “Reproducible Results” section, authors must list which figures, tables, or other results in their manuscript have been reproduced and by whom. This demonstrates that the authors have identified and addressed any bugs, missing materials, unclear directions, and other glitches that often befall initial versions of repositories and prevent reproduction. A “Reproducible Results” section for an initial submission may look like the following:

The Journal has identified multiple new associate editors for reproducibility (AER) to verify statements in the reproducible results sections of submitted manuscripts. On receipt of a manuscript with a reproducible results section, the Journal editor will continue to assign the manuscript to a topical associate editor (AE) for managing the existing peer-review process. The editor will also simultaneously assign the manuscript to an AER. The AER will either attempt to reproduce results themselves or invite a reproducibility reviewer to reproduce manuscript results. The AER or reproducibility reviewer will spend no more than 2 working days to try to reproduce manuscript results. This 2-day limit puts an upper bound on the time commitment requested for an AER or reproducibility reviewer to reproduce results. This limit simultaneously communicates to authors the level of organization needed for data, models, code, directions, and other artifacts so AERs, reproducibility reviewers, and Journal readers can reproduce results in a timely manner even if they are not familiar with the tools or models [see additional suggestions in Rosenberg et al. (2020)].

The AER will complete a short worksheet (available in the Supplemental Materials) that assesses the accuracy of the statements in the manuscript’s “Reproducible Results” section, lists the results that were reproduced, and recommends improvements to make results more reproducible. The AE and AER will coordinate on the subsequent recommendation to the editor on manuscript acceptance. Observations and findings on reproducible results will not typically affect recommendations on manuscript acceptance—only in the extreme case when efforts to reproduce results identify major errors in the manuscript’s overall conclusions. While under review, all data, models, code, directions, and other materials that authors provide will be subject to standard journal confidentiality restrictions (even if the authors permit sharing). Should an anonymous reviewer reproduce some or all results as part of their review, the reviewer may communicate those finding(s) to the AE as part of their review. After manuscript results are successfully reproduced, the authors will add wording like the following to the “Reproducible Results” section of their manuscript:

Fig. 1 shows the new process to handle manuscripts with a new reproducible results section.

By following best practices, authors can include results from open source, proprietary, stochastic, and/or computationally intensive data, models, and code in the “Reproducible Results” section. The editorial board encourages all of these methods along with associated best practices to foster reproducibility. Reproducibility should be agnostic to the tools and technology used. The Journal’s audience is water resource professionals and scientists who may or may not be familiar with a specific software, model, or tool used in the manuscript. Therefore, authors must share code, models, and data, and sufficiently describe and document their use so the AER, a reproducibility reviewer, and Journal readers can identify and follow the steps taken to arrive at the desired results (Rosenberg et al. 2020).

For work that uses a proprietary model or a data-intensive workflow, authors must also

To encourage authors to make their work more reproducible, the Journal will publish technical papers and case studies with verified reproducible results open access free to the authors for the first year, and either free or for a reduced fee in future years, depending on the availability of funds. As part of the review process, the AER will also award a bronze level of reproducibility to each paper that publishes or makes all data, model, code, and directions available online. The AER will additionally award a silver level of reproducibility to papers with verified reproducible results (Stagge et al. 2019). To make it easier to find and track articles with reproducible results, the Journal will feature these articles in a new special collection of articles with reproducible results. Topical AEs will also identify manuscripts that already make all the data, models, and code available online and encourage authors to take the next step of adding a “Reproducible Results” section.

The Journal will also introduce two new annual awards for authors and reproducibility reviewers:

We intend for this new policy to (1) encourage authors to make their results more reproducible; (2) allow scientists and practitioners to more easily find and use reproducible work; (3) encourage further sharing and interaction between authors and readers; (4) recognize and reward authors who make their work more reproducible; and (5) increase the impact of work published in the Journal. We see this new policy as a further step forward to encourage authors, readers, practitioners, and our field to narrow the gap between research and practice. See Rosenberg and Watkins (2018) for further information on the precursor step to prepare a manuscript “Data Availability Statement” section. See Rosenberg et al. (2020) for practices to make results more reproducible.