Free access
Editorial
Oct 29, 2024

Update to the Journal of Water Resources Planning and Management’s Reproducible Results Policy

Publication: Journal of Water Resources Planning and Management
Volume 151, Issue 1
The purpose of this editorial is to update the instructions for authors to prepare and submit a manuscript to the Journal of Water Resources Planning and Management’s opt-in program for reproducible results, launched in 2021 (Rosenberg et al. 2021). This editorial also shares the framework that reproducibility reviewers use to evaluate submissions and provide feedback to authors in order to improve the reproducibility of results published in their manuscript(s) (included as Supplemental Materials). This information is designed to clarify and formalize the process for authors and reviewers, respectively.
Papers accepted into the program will continue to be:
1.
Published Open Access free of author publication charges (a $2,500 value) if the manuscript satisfies the silver badge criteria.
2.
Included in the Special Collection on Articles with Reproducible Results (ASCE 2024).
3.
Given a silver or bronze badge to appear next to the title of the paper. Silver or bronze badges correspond to replicable results and available artifacts, respectively.
4.
Eligible to receive the Journal’s Article Award for Outstanding Efforts to Make Results More Reproducible.
The Journal will also continue to issue an annual Outstanding Efforts to Reproduce Results award. Reproducibility reviewers are eligible for this award.
Find more information about the Journal’s program to reproduce results at https://ascelibrary.org/reprod.

Checklist for Submitting a Manuscript with Reproducible Results

Authors should carefully review this checklist before deciding whether to submit their paper to the Journal’s program to reproduce results.

Step 1: Create a Repository for Data, Model, Code, and Directions

1.
Provide all data, model, code, and directions to reproduce figures, tables, and other results in the manuscript in a publicly accessible repository such as HydroShare, Harvard Dataverse, Figshare, Dryad, Zenodo, or an institutional repository.
2.
Include in the repository a “run all” script or “run all” button that generates all the tables, figures, and other results in the manuscript with one action.
3.
Apply additional best practices to make results reproducible in a repository (see Stagge et al. 2024; Rosenberg et al. 2020).
4.
Note the digital object identifier (DOI) for the repository.
Repositories such as HydroShare, Harvard Dataverse, Figshare, Zenodo, and Dryad automatically provide a DOI.
For GitHub and other repositories that do not automatically provide a DOI, use a service like Zenodo or HydroShare to generate a DOI for your repository (see GitHub 2024).
5.
Ask a colleague not associated with the project to use the materials in the repository to reproduce tables, figures, and other results in the manuscript.

Step 2: Manuscript Information

1.
Provide a “Data Availability Statement” section after the conclusions (Rosenberg and Watkins 2018).
Add the text “The following data, models, or code generated or used during the study are available in a repository or online.”
Cite the repository where the data, model, code, and directions are available (see Step 1: Create a Repository for Data, Model, Code, and Directions).
Add the citation to the manuscript references and include the DOI generated in Step 1.
2.
Provide a “Reproducible Results” section immediately after the “Data Availability Statement” section. The text should state who reproduced results (the person the authors asked in Step 1), their affiliation, and what results they reproduced. Example from Cordeiro et al. (2021): Soraia Pereira (Universidade de Lisboa) downloaded all the materials, installed, ran the models using the data and functions in “Workspace_data_functions.RData”, and reproduced the results in Tables 1 and 2.

Step 3. Submitting Your Paper in Editorial Manager

1.
When submitting your paper via the Editorial Manager site, select option e: “Some or all data, models, or code generated or used during the study are available in a repository online in accordance with funder data retention policies” for the data availability question in the “Additional Information” section of the form. This will trigger a new question, “Do you want to participate in the reproducible results program, and does your manuscript meet the criteria above?” Select Yes.
Note: authors must still provide all data, models, code, and directions in their repository to be eligible for the journal’s program to reproduce results.
2.
A new box will appear with the text “Please provide a direct link to the data, if applicable.” Enter the DOI created in Step 1: Create a Repository for Data, Model, Code, and Directions.
3.
Complete the remaining submission steps.

Step 4. Peer Review and Reproducibility

1.
The journal editor will assign the manuscript to an associate editor for reproducibility (AER).
2.
The AER will invite a reproducibility reviewer to attempt to reproduce the results in the manuscript.
3.
Simultaneously, the manuscript will undergo the typical peer review process for scientific content.
4.
The reproducibility reviewer or AER will use the “Form to Assess Result Reproducibility of Manuscripts” (see Supplemental Materials) to assess reproducibility and provide authors feedback to improve the reproducibility of the results in their manuscript.

Step 5. Acceptance into the Reproducible Results Program

1.
The authors will receive a letter of acceptance. Within the acceptance letter the journal editor will:
Provide the authors with a new statement saying who from the journal reproduced results and which results they reproduced. For example: Jane Smith (reproducibility reviewer) downloaded the data, models, code, and directions from (citation) and reproduced Figs. 1 and 2 and Table 3 in the manuscript.
Ask the authors to replace their original sentence in the “Reproducible Results” section of their manuscript with this new sentence and update the manuscript in Editorial Manager during creation of the final proof before publication.
Share the expectation that each author of the paper completes a reproducibility review for the journal in the future.
2.
If the authors receive the silver badge, their article will be published Open Access without author publication charges. This step is managed internally by the journal.

Step 6. Additional Actions upon Publication of the Paper

1.
The Journal will add the paper to the Special Collection on Articles with Reproducible Results (ASCE 2024).
2.
The Journal will add a badge to appear online next to the title of the paper. The paper will receive a silver badge if the reviewers were able to replicate results. The paper will receive a bronze badge if all artifacts (data, code, models, instructions) were provided, but reviewers could not replicate results. Bronze and silver reproducibility badges signify the authors’ demonstrated commitment to share knowledge widely and accessibly in order to advance the field.
3.
The authors of the paper will be eligible to receive the Journal’s annual Article Award for outstanding effort to make results more reproducible.

Step 7. Participate as a Reproducibility Reviewer

1.
Reproducibility badges reflect significant effort by reproducibility reviewers and editors who volunteer their time to verify the authors’ stated reproducibility.
2.
Kindly note that submitting a manuscript to the reproducible results program constitutes a pledge by all manuscript authors to assess reproducibility of at least one submission in the future should their paper be accepted. This helps the program remain sustainable for future authors.

Find Further Information on the Journal’s Reproducibility Webpage

The journal’s reproducibility resource page is online in the ASCE Library (https://ascelibrary.org/reprod) and includes:
A 2024 update to Journal’s results reproducible policy (Stagge et al. 2024).
Program accomplishments, lessons, and next steps to make research more reproducible (Stagge et al. 2024).
More than 40 tips to make research more reproducible (Rosenberg et al. 2020).
A link to a collection of papers with verified reproduced results.
A list of articles that received the annual award for outstanding effort to make results more reproducible.
A list of reproducibility reviewers who received the annual award for outstanding effort to reproduce results.

Supplemental Materials

File (supplemental materials_jwrmd5.wreng-6726_stagge.pdf)

References

ASCE. 2024. “Special collection on articles with reproducible results.” Accessed September 6, 2024. https://ascelibrary.org/jwrmd5/reproducibility_papers.
Cordeiro, C., A. Borges, and M. Rosário Ramos. 2021. “A strategy to assess water meter performance.” J. Water Resour. Plann. Manage. 148 (2): 05021027. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001492.
GitHub. 2024. “Referencing and citing content: You can use third-party tools to cite and reference content on GitHub.” GitHub Documents. Accessed September 6, 2024. https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content.
Rosenberg, D. E., Y. Filion, R. Teasley, S. Sandoval-Solis, J. S. Hecht, J. E. van Zyl, G. F. McMahon, J. S. Horsburgh, J. R. Kasprzyk, and D. G. Tarboton. 2020. “The next frontier: Making research more reproducible.” J. Water Resour. Plann. Manage. 146 (6): 01820002. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001215.
Rosenberg, D. E., A. S. Jones, Y. Filion, R. Teasley, S. Sandoval-Solis, J. H. Stagge, A. Abdallah, A. Castronova, A. Ostfeld, and D. Watkins Jr. 2021. “Reproducible results policy.” J. Water Resour. Plann. Manage. 147 (2): 01620001. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001368.
Rosenberg, D. E., and D. W. Watkins Jr. 2018. “New policy to specify availability of data, models, and code.” J. Water Resour. Plann. Manage. 144 (9): 01618001. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000998.
Stagge, J. H., D. E. Rosenberg, A. M. Castronova, A. Ostfeld, and A. Spackman Jones. 2024. “Journal of Water Resources Planning and Management’s reproducibility review program: Accomplishments, lessons, and next steps.” J. Water Resour. Plann. Manage. 150 (8): 01824001. https://doi.org/10.1061/JWRMD5.WRENG-6559.

Information & Authors

Information

Published In

Go to Journal of Water Resources Planning and Management
Journal of Water Resources Planning and Management
Volume 151Issue 1January 2025

History

Received: Jun 26, 2024
Accepted: Jun 27, 2024
Published online: Oct 29, 2024
Published in print: Jan 1, 2025
Discussion open until: Mar 29, 2025

Permissions

Request permissions for this article.

Authors

Affiliations

Assistant Professor, Dept. of Civil, Environmental and Geodetic Engineering, Ohio State Univ., Columbus, OH 43210 (corresponding author). ORCID: https://orcid.org/0000-0002-3667-2904. Email: [email protected]
Professor, Civil and Environmental Engineering Dept., Utah State Univ., Logan, UT 84322. ORCID: https://orcid.org/0000-0003-2163-2907. Email: [email protected]
Anthony M. Castronova [email protected]
Senior Research Scientist, Consortium of Universities for the Advancement of Hydrologic Science, Inc., 1167 Massachusetts Ave., Suite 419, Arlington, MA 02476. Email: [email protected]
Avi Ostfeld, F.ASCE [email protected]
Professor, Dept. of Civil and Environmental Engineering, Technion–Israel Institute of Technology, Haifa 32000, Israel. Email: [email protected]
Amber Spackman Jones, A.M.ASCE [email protected]
Research Associate, Civil and Environmental Engineering Dept., Utah State Univ., Logan, UT 84322. Email: [email protected]

Metrics & Citations

Metrics

Citations

Download citation

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

View Options

Media

Figures

Other

Tables

Share

Share

Copy the content Link

Share with email

Email a colleague

Share