Journal of Water Resources Planning and Management’s Reproducibility Review Program: Accomplishments, Lessons, and Next Steps

In November 2020, the editors of the Journal of Water Resources Planning and Management (JWRPM) launched a Reproducibility Review Program (Rosenberg et al. 2021). This initiative was inspired by studies showing a lack of reproducibility in water-related journals (Stagge et al. 2019), calls for better reviews of claims of reproducibility (Goodman et al. 2016), and similar review programs at journals in other domains (Rosenberg et al. 2021; Rosenberg and Watkins 2018). JWRPM’s voluntary program incentivizes authors to publish data, models, code, and directions with their articles so that an independent reviewer can replicate part or all of the authors’ work. The goal of the JWRPM Reproducibility Review Program is to promote a cultural shift toward making research more accessible and reproducible, thereby accelerating science and increasing impact. The program has five objectives (Rosenberg et al. 2021):

For more details, visit the Reproducibility Hub (https://ascelibrary.org/reprod) for products of the Reproducibility Review Program, including:

In this editorial, we—JWRPM’s associate editors for reproducibility (AERs)—share program accomplishments, challenges and lessons learned, and next steps to expand the pilot program to other ASCE journals. We also share our experience as a potential model to foster more reproducible and open research products, set forth as a goal by research sponsors and government agencies (Burgelman et al. 2019; European Commission et al. 2020; Nelson 2022; National Science Foundation and Institute of Education Sciences, US Department of Education 2018).

JWRPM’s Reproducibility Review Program is already successfully addressing its five initial objectives, and the program continues to build momentum. Motivating authors to make their work more reproducible (Objective 1) can be measured by participation in the program (Fig. 1). Since its inception, approximately 3 years ago in November 2020, the program has awarded 10 Silver badges for manuscripts where the reviewers could reproduce all or part of the study’s results, and three Bronze badges for manuscripts that provided research artifacts (Table 1). An additional 11 manuscripts are currently in various stages of reproducibility review. Furthermore, the authors seven manuscripts elected to withdraw from the Reproducibility Review Program after an initial reproducibility review, but these manuscripts were ultimately published as traditional articles. Excluding manuscripts still in review or declined for technical reasons, 65% (13 of 20 manuscripts) were successfully reproduced by reviewers. These manuscripts are not representative of all JWRPM manuscripts because the authors self-selected by applying to the voluntary program. From November 2020 to July 2023, 557 articles were published by JWRPM, making the 13 reproduced articles an exclusive group (Bastidas Pacheco et al. 2023; Cordeiro et al. 2022; Hadjimichael et al. 2023; Jander et al. 2023; Jaramillo and Saldarriaga 2023; Morgan and Lane 2022; Obringer et al. 2022; Rasmussen et al. 2023; Rodríguez-Martínez et al. 2023; Thomas and Sela 2023; Tran et al. 2023; Vrachimis et al. 2022; Wang and Rosenberg 2023). We intend to further increase the proportion of JWRPM articles successfully handled by the JWRPM Reproducibility Review Program.

A major motivator for authors to participate in the program is the offer of free open access without author publishing charges (APCs) for papers receiving the Silver reproducibility badge—a significant $2,500 savings as of this writing. ASCE Publications and the Environmental and Water Resources Institute (EWRI) donated $40,000 and $20,000, respectively, to support free open access publishing. The 11 papers that met the Silver badge reproducibility criteria were awarded free open access publication. Offering open access publication with no APCs helps meet Objective 4, recognize and reward authors who make their work more reproducible. Furthermore, JWRPM implemented annual awards to recognize and reward authors (Outstanding Effort to Make Results More Reproducible) as well as editors (Outstanding Effort to Reproduce Results). Papers that meet the Silver and Bronze criteria are awarded reproducibility badges, and JWRPM created a special collection page designed to drive readers to reproduced papers. Increased research visibility helps reward authors who make additional efforts for reproducibility.

Objectives 2, 3, and 5 all describe increased research impact in various forms. Three years is too soon to detect an increase in impact in citations or whether new research is building on successfully reproduced manuscripts. In the coming years, we foresee an increased impact by publishing reproduced papers as open access because papers published open access are downloaded and cited at higher rates across multiple fields (McCabe and Snyder 2014; Ottaviani 2016; Piwowar et al. 2007, 2018).

ASCE intends to expand the JWRPM Reproducibility Review Program to other ASCE journals. This section presents some challenges and lessons learned from JWRPM’s experience to demonstrate how other journals could implement similar programs.

JWRPM intends to proceed with the Reproducibility Review Program, and editors will continue to advocate for the program. We aim to expand the number of published papers recognized for reproducibility. We also plan to hold training workshops to educate both potential reviewers and authors.

Currently, JWRPM is using external funding to publish successfully reproduced manuscripts as open access without author publishing charges. We hope to develop new funding sources and funding models to support this popular motivator. To our knowledge, JWRPM is the only journal with a business model that offers free open access as an incentive for successfully reproduced manuscripts. The journals ReScience C and ReScience X also offer open access and fully reproduced research, but they do not publish new research with an internal reproducibility review, instead publishing computational reproductions of studies published elsewhere. We commend ASCE for supporting this initiative and for plans to expand it to other journals in the ASCE portfolio. We encourage other academic journals and supporting organizations to adopt similar practices.

Our goal with this program continues to be to improve reproducibility within the water resources field. We have set and are achieving clear and attainable benchmarks. We are highlighting and rewarding authors that rise to the challenge to make their work reproducible. Papers with reproduced results are patterning best practices and helping shift our science and engineering culture toward reproducibility as the default when conducting analyses and submitting a manuscript.

All data for this editorial are included in a Zenodo repository (Stagge et al. 2023). The code used to generate Fig. 1 is available in a Github repository (Stagge et al. 2023).

Kyungmin Sung and Irenee Munyejuru (Ohio State University) downloaded the Github repository, ran the code, and successfully reproduced Fig. 1 as presented here.