Forecast Families: A New Method to Systematically Evaluate the Benefits of Improving the Skill of an Existing Forecast
Publication: Journal of Water Resources Planning and Management
Volume 149, Issue 5
Abstract
A growing number of studies have investigated how forecast skill, i.e., predictive power, translates into forecast value, i.e., usefulness, for improving forecast-informed decisions. The relationship between skill and value is widely understood to be complex and case-specific, yet few methods enable its systematic exploration using realistic forecast errors. This paper addresses this gap by proposing a single-parameter linear scaling method to generate families of synthetic forecasts with the desired skill improvements on an existing hindcast (a retrospective forecast of already-observed events). The method is applicable to any quantity for which a deterministic or ensemble hindcast is available, and generates a set of forecasts with different skill but strictly proportional errors. This like-for-like comparison preserves the autocorrelation and cross-correlations of errors, and opens the door for thorough, yet easily interpretable, explorations of the relationship between skill and value of a realistic forecast. We apply this new method to seasonal precipitation hindcasts (produced by the fifth generation of the Seasonal forecasting System of the European Centre for Medium-range Weather Forecasts, ECMWF-SEAS5) in order to explore their value for improving the management of a water supply system in the UK. The application showed that although value generally increases with skill, the skill–value relationship is not necessarily linear, and it strongly depends on operational preferences and hydrological conditions (wet or dry years). It also suggests that the forecast families methodology can help water managers and forecast developers identify when a skill increase would be most valuable. This has the potential to foster productive two-way conversations between forecast producers and users.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
ECMWF hindcast are available under a range of licences (Vitart et al. 2017). For more information please visit https://apps.ecmwf.int/datasets/data/s2s-reforecasts-instantaneous-accum-ecmf/levtype=sfc/type=cf/. The code used for preprocessing ECMWF hindcast and implementing the simulation-optimization methodology is available at https://ironstoolbox.github.io/ (Peñuela et al. 2021). The reservoir system data of the case study are property of Wessex Water Ltd. and as such cannot be shared by the authors. The code for generating forecast families is available in the Zenodo open-access repository at https://doi.org/10.5281/zenodo.7327755 along with a demonstration including figures in the paper illustrating forecast family generation.
Acknowledgments
Charles Rougé is partially funded by the UK National Environmental Research Council (NERC) via a UK Climate Resilience Embedded Researcher Grant (NE/V010239/1). Andres Peñuela is funded by the Spanish Ministry of Science and Innovation, the Spanish State Research Agency, through the Severo Ochoa and María de Maeztu Program for Centers and Units of Excellence in Research and Development (CEX2019-000968-M). Francesca Pianosi is partially funded by the UK Engineering and Physical Sciences Research Council (EPSRC) via an Early Career Fellowship (EP/R007330/1). For the purpose of open access, authors have applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising. The authors are also very grateful to Wessex Water for the data provided. The authors wish to thank the Copernicus Climate Change and Atmosphere Monitoring Services for providing the seasonal forecasts generated by the ECMWF seasonal forecasting systems (SEAS5). Neither the European Commission nor ECMWF is responsible for any use that may be made of the Copernicus information or data it contains. Finally, authors would like to thank the Editor, Associate Editor, and three anonymous reviewers for their comments, which greatly improved this paper.
References
Alley, R. B., K. A. Emanuel, and F. Zhang. 2019. “Advances in weather prediction.” Science 363 (6425): 342–344. https://doi.org/10.1126/science.aav7274.
Anghileri, D., S. Monhart, C. Zhou, K. Bogner, A. Castelletti, P. Burlando, and M. Zappa. 2019. “The value of subseasonal hydrometeorological forecasts to hydropower operations: How much does preprocessing matter?” Water Resour. Res. 55 (12): 10159–10178. https://doi.org/10.1029/2019WR025280.
Anghileri, D., N. Voisin, A. Castelletti, F. Pianosi, B. Nijssen, and D. P. Lettenmaier. 2016. “Value of long-term streamflow forecasts to reservoir operations for water supply in snow-dominated river catchments.” Water Resour. Res. 52 (6): 4209–4225. https://doi.org/10.1002/2015WR017864.
Arsenault, R., and P. Côté. 2019. “Analysis of the effects of biases in ensemble streamflow prediction (ESP) forecasts on electricity production in hydropower reservoir management.” Hydrol. Earth Syst. Sci. 23 (6): 2735–2750. https://doi.org/10.5194/hess-23-2735-2019.
Bauer, P., A. Thorpe, and G. Brunet. 2015. “The quiet revolution of numerical weather prediction.” Nature 525 (7567): 47–55. https://doi.org/10.1038/nature14956.
Bergström, S. 1995. “The HBV model.” In Computer models of watershed hydrology, edited by V. Singh, 443–476. Highlands Ranch, CO: Water Resources Publications.
Blank, J., and K. Deb. 2020. “Pymoo: Multi-objective optimization in python.” IEEE Access 8 (Apr): 89497–89509. https://doi.org/10.1109/ACCESS.2020.2990567.
Brodeur, Z. P., and S. Steinschneider. 2021. “A multivariate approach to generate synthetic short-to-medium range hydro-meteorological forecasts across locations, variables, and lead times.” Water Resour. Res. 57 (6): e2020WR029453. https://doi.org/10.1029/2020WR029453.
Bruno Soares, M., M. Alexander, and S. Dessai. 2018. “Sectoral use of climate information in Europe: A synoptic overview.” Clim. Serv. 9 (Mar): 5–20. https://doi.org/10.1016/j.cliser.2017.06.001.
Bruno Soares, M., and S. Dessai. 2016. “Barriers and enablers to the use of seasonal climate forecasts amongst organisations in Europe.” Clim. Change 137 (1–2): 89–103. https://doi.org/10.1007/s10584-016-1671-8.
Cassagnole, M., M.-H. Ramos, I. Zalachori, G. Thirel, R. Garçon, J. Gailhard, and T. Ouillon. 2021. “Impact of the quality of hydrological forecasts on the management and revenue of hydroelectric reservoirs—A conceptual approach.” Hydrol. Earth Syst. Sci. 25 (2): 1033–1052. https://doi.org/10.5194/hess-25-1033-2021.
Cohen, J., D. Coumou, J. Hwang, L. Mackey, P. Orenstein, S. Totz, and E. Tziperman. 2019. “S2s reboot: An argument for greater inclusion of machine learning in subseasonal to seasonal forecasts.” WIREs Clim. Change 10 (2): e00567. https://doi.org/10.1002/wcc.567.
Crochemore, L., M.-H. Ramos, and I. G. Pechlivanidis. 2020. “Can continental models convey useful seasonal hydrologic information at the catchment scale?” Water Resour. Res. 56 (2): e2019WR025700. https://doi.org/10.1029/2019WR025700.
Doering, K., J. Quinn, P. M. Reed, and S. Steinschneider. 2021. “Diagnosing the time-varying value of forecasts in multiobjective reservoir control.” J. Water Resour. Plann. Manage. 147 (7): 04021031. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001386.
Ficchì, A., L. Raso, D. Dorchies, F. Pianosi, P.-O. Malaterre, P.-J. V. Overloop, and M. Jay-Allemand. 2016. “Optimal operation of the multireservoir system in the seine river basin using deterministic and ensemble forecasts.” J. Water Resour. Plann. Manage. 142 (1): 05015005. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000571.
Fleming, S. W., D. C. Garen, A. G. Goodbody, C. S. McCarthy, and L. C. Landers. 2021. “Assessing the new natural resources conservation service water supply forecast model for the American West: A challenging test of explainable, automated, ensemble artificial intelligence.” J. Hydrol. 602 (Nov): 126782. https://doi.org/10.1016/j.jhydrol.2021.126782.
Fortin, V., A.-C. Favre, and M. Saïd. 2006. “Probabilistic forecasting from ensemble prediction systems: Improving upon the best-member method by using a different weight and dressing kernel for each member.” Q. J. R. Meteorol. Soc. 132 (617): 1349–1369. https://doi.org/10.1256/qj.05.167.
Giuliani, M., L. Crochemore, I. Pechlivanidis, and A. Castelletti. 2020. “From skill to value: Isolating the influence of end user behavior on seasonal forecast assessment.” Hydrol. Earth Syst. Sci. 24 (12): 5891–5902. https://doi.org/10.5194/hess-24-5891-2020.
Hersbach, H. 2000. “Decomposition of the continuous ranked probability score for ensemble prediction systems.” Weather Forecasting 15 (5): 559–570. https://doi.org/10.1175/1520-0434(2000)015%3C0559:DOTCRP%3E2.0.CO;2.
Jackson-Blake, L. A., et al. 2022. “Opportunities for seasonal forecasting to support water management outside the tropics.” Hydrol. Earth Syst. Sci. 26 (5): 1389–1406. https://doi.org/10.5194/hess-26-1389-2022.
Johnson, S. J., et al. 2019. “Seas5: The new ECMWF seasonal forecast system.” Geosci. Model Dev. 12 (3): 1087–1117. https://doi.org/10.5194/gmd-12-1087-2019.
Lamontagne, J. R., and J. R. Stedinger. 2018. “Generating synthetic streamflow forecasts with specified precision.” J. Water Resour. Plann. Manage. 144 (4): 04018007. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000915.
Maurer, E. P., and D. P. Lettenmaier. 2004. “Potential effects of long-lead hydrologic predictability on Missouri river main-stem reservoirs.” J. Clim. 17 (1): 174–186. https://doi.org/10.1175/1520-0442(2004)017%3C0174:PEOLHP%3E2.0.CO;2.
Nayak, M. A., J. D. Herman, and S. Steinschneider. 2018. “Balancing flood risk and water supply in California: Policy search integrating short-term forecast ensembles with conjunctive use.” Water Resour. Res. 54 (10): 7557–7576. https://doi.org/10.1029/2018WR023177.
Pappenberger, F., M. Ramos, H. Cloke, F. Wetterhall, L. Alfieri, K. Bogner, A. Mueller, and P. Salamon. 2015. “How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction.” J. Hydrol. 522 (Mar): 697–713. https://doi.org/10.1016/j.jhydrol.2015.01.024.
Peñuela, A., C. Hutton, and F. Pianosi. 2020. “Assessing the value of seasonal hydrological forecasts for improving water resource management: Insights from a pilot application in the UK.” Hydrol. Earth Syst. Sci. 24 (12): 6059–6073. https://doi.org/10.5194/hess-24-6059-2020.
Peñuela, A., C. Hutton, and F. Pianosi. 2021. “An open-source package with interactive Jupyter Notebooks to enhance the accessibility of reservoir operations simulation and optimization.” Environ. Modell. Software 145 (Nov): 105188. https://doi.org/10.1016/j.envsoft.2021.105188.
Pereira, M. V. F., G. C. Oliveira, C. C. G. Costa, and J. Kelman. 1984. “Stochastic streamflow models for hydroelectric systems.” Water Resour. Res. 20 (3): 379–390. https://doi.org/10.1029/WR020i003p00379.
Rougé, C., P. M. Reed, D. S. Grogan, S. Zuidema, A. Prusevich, S. Glidden, J. R. Lamontagne, and R. B. Lammers. 2021. “Coordination and control—Limits in standard representations of multi-reservoir operations in hydrological modeling.” Hydrol. Earth Syst. Sci. 25 (3): 1365–1388. https://doi.org/10.5194/hess-25-1365-2021.
Roulston, M., and L. Smith. 2003. “Combining dynamical and statistical ensembles.” Tellus A 55 (1): 16–30. https://doi.org/10.3402/tellusa.v55i1.12082.
Saltelli, A., M. Ratto, T. Andres, F. Campolongo, J. Cariboni, D. Gatelli, M. Saisana, and S. Tarantola. 2008. Global sensitivity analysis: The primer. New York: Wiley.
Sankarasubramanian, A., U. Lall, F. A. Souza Filho, and A. Sharma. 2009. “Improved water allocation utilizing probabilistic climate forecasts: Short-term water contracts in a risk management framework.” Water Resour. Res. 45 (11). https://doi.org/10.1029/2009WR007821.
Slater, L. J., and G. Villarini. 2018. “Enhancing the predictability of seasonal streamflow with a statistical-dynamical approach.” Geophys. Res. Lett. 45 (13): 6504–6513. https://doi.org/10.1029/2018GL077945.
Stockdale, T., et al. 2018. Seas5 and the future evolution of the long-range forecast system. Reading, UK: European Centre for Medium-Range Weather Forecasts.
Turner, S. W. D., J. C. Bennett, D. E. Robertson, and S. Galelli. 2017. “Complex relationship between seasonal streamflow forecast skill and value in reservoir operations.” Hydrol. Earth Syst. Sci. 21 (9): 4841–4859. https://doi.org/10.5194/hess-21-4841-2017.
Turner, S. W. D., W. Xu, and N. Voisin. 2020. “Inferred inflow forecast horizons guiding reservoir release decisions across the United States.” Hydrol. Earth Syst. Sci. 24 (3): 1275–1291. https://doi.org/10.5194/hess-24-1275-2020.
Vitart, F., et al. 2017. “The subseasonal to seasonal (S2S) prediction project database.” Bull. Am. Meteorol. Soc. 98 (1): 163–173. https://doi.org/10.1175/BAMS-D-16-0017.1.
Whateley, S., R. N. Palmer, and C. Brown. 2015. “Seasonal hydroclimatic forecasts as innovations and the challenges of adoption by water managers.” J. Water Resour. Plann. Manage. 141 (5): 04014071. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000466.
Yang, G., S. Guo, P. Liu, and P. Block. 2020. “Integration and evaluation of forecast-informed multiobjective reservoir operations.” J. Water Resour. Plann. Manage. 146 (6): 04020038. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001229.
Zhao, T., X. Cai, and D. Yang. 2011. “Effect of streamflow forecast uncertainty on real-time reservoir operation.” Adv. Water Resour. 34 (4): 495–504. https://doi.org/10.1016/j.advwatres.2011.01.004.
Zhao, T., and J. Zhao. 2014. “Forecast-skill-based simulation of streamflow forecasts.” Adv. Water Resour. 71 (Sep): 55–64. https://doi.org/10.1016/j.advwatres.2014.05.011.
Zhao, T., J. Zhao, D. Yang, and H. Wang. 2013. “Generalized martingale model of the uncertainty evolution of streamflow forecasts.” Adv. Water Resour. 57 (Jul): 41–51. https://doi.org/10.1016/j.advwatres.2013.03.008.
Information & Authors
Information
Published In
Journal of Water Resources Planning and Management
Volume 149 • Issue 5 • May 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Aug 1, 2022
Accepted: Jan 5, 2023
Published online: Mar 9, 2023
Published in print: May 1, 2023
Discussion open until: Aug 9, 2023
ASCE Technical Topics:
- Business management
- Climates
- Engineering fundamentals
- Environmental engineering
- Errors (statistics)
- Forecasting
- Fouling
- Linear functions
- Management methods
- Mathematical functions
- Mathematics
- Parameters (statistics)
- Practice and Profession
- Seasonal variations
- Statistics
- Waste management
- Waste treatment
- Water and water resources
- Water management
- Water supply
- Water supply systems
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.