Operational Comparison of Rainfall-Runoff Models through Hypothesis Testing
Publication: Journal of Hydrologic Engineering
Volume 25, Issue 4
Abstract
Assessing rainfall-runoff model performance and selecting the model best suited are important considerations in operational hydrology. However, model choice often is heuristic and based on a simplistic comparison of a single performance criterion without considering the statistical significance of differences in performance. This potentially is problematic because interpretation of a single performance criterion is subjective to the user. This paper removed the subjectivity by applying a jackknife split-sample calibration method to create a sample mean of performance for competing models which are used in a paired -test, allowing statements of statistical significance to be made. A second method was presented based on a hypothesis test in the binomial distribution, considering model performance across a group of catchments. A case study comparing the performance of two rainfall-runoff models across 27 urban catchments within the Thames basin showed that although the urban signal was difficult to detect on single catchment, it was significant across the group of catchments depending upon the choice of performance criteria. These results demonstrated the operational applicability of the new tools and the benefits of considering model performance in a probabilistic framework.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
The following data used during the study were provided by a third-party: rainfall data (Tanguy et al. 2014), evaporation data (Robinson et al. 2016), and river flow data (NRFA 2018). Direct requests for these materials may be made to the provider as indicated in the “Acknowledgments.”
The following models and code generated used during the study are available from the corresponding author by request: URMOD (hydrological model), and jackknife calibration/validation code.
Acknowledgments
The authors thank the three anonymous reviewers for helpful comments which helped improve the paper, and the National River Flow Archive (NRFA) for providing access to hydrological data and catchment shapefiles. Funding from the Engineering and Physical Sciences Research Council (EPSEC) (Grant No. 1552086) is acknowledged. The authors also thank the NERC-funded POLLCURB project for providing access to the hydrological and land-use data used in this study (NE/K002317/1).
References
Addor, N., and L. Melsen. 2019. “Legacy, rather than adequacy, drives the selection of hydrological models.” Water Resour. Res. 55 (1): 378–390. https://doi.org/10.1029/2018WR022958.
Andréassian, V., C. Perrin, L. Berthet, N. Le Moine, J. Lerat, C. Loumagne, L. Oudin, T. Mathevet, M. Ramos, and A. Valéry. 2009. “Crash tests for a standardized evaluation of hydrological models.” Hydrol. Earth Syst. Sci. 13 (10): 1757–1764. https://doi.org/10.5194/hess-13-1757-2009.
Anh, N. L., J. Boxall, A. Saul, and P. Willems. 2010. “An evaluation of three lumped conceptual rainfall-runoff models at catchment scale.” In Proc., 3rd Int. Symp. on British Hydrological Society. London: British Hydrological Society.
Bayliss, A., K. Black, A. Fava-Verde, and T. Kjeldsen. 2006. URBEXT2000: A new FEH catchment descriptor. calculation, dissemination and application. Wallingford, UK: Dept. for Environment Food and Rural Affairs, CEH Wallingford.
Beven, K. J. 2011. Rainfall-runoff modelling: The primer. Chichester, UK: Wiley.
Bouffard, J.-S. 2014. “A comparison of conceptual rainfall-runoff modelling structures and approaches for hydrologic prediction in ungauged peatland basins of the James Bay Lowlands.” Ph.D. thesis, Dept. of Geography and Environmental Studies, Carleton Univ. https://curve.carleton.ca/7ed3b15f-bfff-4027-9a87-f314602d7a1a.
Coron, L., V. Andreassian, C. Perrin, J. Lerat, J. Vaze, M. Bourqui, and F. Hendrickx. 2012. “Crash testing hydrological models in contrasted climate conditions: An experiment on 216 Australian catchments.” Water Resour. Res. 48 (5): W05552. https://doi.org/10.1029/2011WR011721.
Criss, R. E., and W. E. Winston. 2008. “Do Nash values have value? Discussion and alternate proposals.” Hydrol. Process. 22 (14): 2723–2725. https://doi.org/10.1002/hyp.7072.
Donnelly-Makowecki, L., and R. Moore. 1999. “Hierarchical testing of three rainfall–runoff models in small forested catchments.” J. Hydrol. 219 (3): 136–152. https://doi.org/10.1016/S0022-1694(99)00056-6.
Duan, Q., V. K. Gupta, and S. Sorooshian. 1993. “Shuffled complex evolution approach for effective and efficient global minimization.” J. Optim. Theory Appl. 76 (3): 501–521. https://doi.org/10.1007/BF00939380.
Efron, B. 1982. Vol. 38 of The jackknife, the bootstrap, and other resampling plans. Philadelphia: Society for Industrial and Applied Mathematics.
Ewen, J. 2011. “Hydrograph matching method for measuring model performance.” J. Hydrol. 408 (1): 178–187. https://doi.org/10.1016/j.jhydrol.2011.07.038.
Ewen, J., and G. O’Donnell. 2012. “Prediction intervals for rainfall–runoff models: Raw error method and split-sample validation.” Hydrol. Res. 43 (5): 637–648. https://doi.org/10.2166/nh.2012.038.
Fidal, J. 2019. “Investigating the impact of urbanisation on rainfall-runoff models.” Ph.D. thesis, Dept. of Architecture and Civil Engineering, Univ. of Bath.
Fleming, S. 2009. “An informal survey of watershed model users: Preferences, applications, and rationales.” Streamline Watershed ManageBull 13 (1): 32–35.
Gharari, S., M. Hrachowitz, F. Fenicia, and H. Savenije. 2013. “An approach to identify time consistent model parameters: Sub-period calibration.” Hydrol. Earth Syst. Sci. 17 (1): 149–161. https://doi.org/10.5194/hess-17-149-2013.
Gupta, H. V., H. Kling, K. K. Yilmaz, and G. F. Martinez. 2009. “Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling.” J. Hydrol. 377 (1–2): 80–91. https://doi.org/10.1016/j.jhydrol.2009.08.003.
Jones, D. A., and A. L. Kay. 2007. “Uncertainty analysis for estimating flood frequencies for ungauged catchments using rainfall-runoff models.” Adv. Water Resour. 30 (5): 1190–1204. https://doi.org/10.1016/j.advwatres.2006.10.009.
Kavetski, D., G. Kuczera, and S. W. Franks. 2006. “Bayesian analysis of input uncertainty in hydrological modeling. 1: Theory.” Water Resour. Res. 42 (3): W03407. https://doi.org/10.1029/2005WR004368.
Kirchner, J. W., R. P. Hooper, C. Kendall, C. Neal, and G. Leavesley. 1996. “Testing and validating environmental models.” Sci. Total Environ. 183 (1–2): 33–47. https://doi.org/10.1016/0048-9697(95)04971-1.
Kjeldsen, T., E. Stewart, J. Packman, S. Folwell, and A. Bayliss. 2005. Revitalisation of the FSR/FEH rainfall-runoff method. Wallingford, UK: CEH Wallingford.
Klemeš, V. 1986. “Operational testing of hydrological simulation models.” Hydrol. Sci. J. 31 (1): 13–24. https://doi.org/10.1080/02626668609491024.
Kohavi, R. 1995. “A study of cross-validation and bootstrap for accuracy estimation and model selection.” In Vol. 2 of Proc., 14th Int. Joint Conf. on Artificial Intelligence IjcaI’95, 1137–1145. Burlington, MA: Morgan Kaufmann.
Krause, P., D. P. Boyle, and F. Bäse. 2005. “Comparison of different efficiency criteria for hydrological model assessment.” Adv. Geosci. 5: 89–97. https://doi.org/10.5194/adgeo-5-89-2005.
Legates, D. R., and G. J. McCabe. 1999. “Evaluating the use of goodness-of-fit measures in hydrologic and hydroclimatic model validation.” Water Resour. Res. 35 (1): 233–241. https://doi.org/10.1029/1998WR900018.
Mishra, S. 2009. “Uncertainty and sensitivity analysis techniques for hydrologic modeling.” J. Hydroinf. 11 (3–4): 282–296. https://doi.org/10.2166/hydro.2009.048.
Nash, J. E., and J. V. Sutcliffe. 1970. “River flow forecasting through conceptual models. Part I: A discussion of principles.” J. Hydrol. 10 (3): 282–290. https://doi.org/10.1016/0022-1694(70)90255-6.
NRFA (National River Flow Archive). 2018. “National river flow archive.” NERC CEH, Wallingford. Accessed March 18, 2015. https://nrfa.ceh.ac.uk/.
Pappenberger, F., and K. J. Beven. 2006. “Ignorance is bliss: Or seven reasons not to use uncertainty analysis.” Water Resour. Res. 42 (5): W05302. https://doi.org/10.1029/2005WR004820.
Pechlivanidis, I. G., B. Jackson, and H. McMillan. 2010. “The use of entropy as a model diagnostic in rainfall-runoff modelling.” In Proc., 5th Biennial Meeting of Int. Environmental Modelling and Software Society (iEMSs), edited by D. A. Swayne, W. Yang, A. A. Voinov, A. Rizzoli, and T. Filatova. Provo, UT: Brigham Young Univ.
Quenouille, M. H. 1956. “Notes on bias in estimation.” Biometrika 43 (3/4): 353–360. https://doi.org/10.2307/2332914.
Refsgaard, J. C. 1997. “Parameterisation, calibration and validation of distributed hydrological models.” J. Hydrol. 198 (1): 69–97. https://doi.org/10.1016/S0022-1694(96)03329-X.
Refsgaard, J. C., and J. Knudsen. 1996. “Operational validation and intercomparison of different types of hydrological models.” Water Resour. Res. 32 (7): 2189–2202. https://doi.org/10.1029/96WR00896.
Robinson, E. L., E. Blyth, D. B. Clark, E. Comyn-Platt, J. Finch, and A. C. Rudd. 2016. “Climate hydrology and ecology research support system potential evapotranspiration dataset for Great Britain (1961-2015) [CHESS-PE].” Accessed January 27, 2020. https://doi.org/10.5285/8baf805d-39ce-4dac-b224-c926ada353b7.
Santos, C., C. Almeida, T. Ramos, F. Rocha, R. Oliveira, and R. Neves. 2018. “Using a hierarchical approach to calibrate swat and predict the semi-arid hydrologic regime of northeastern brazil.” Water 10 (9): 1137. https://doi.org/10.3390/w10091137.
Schaefli, B., and H. V. Gupta. 2007. “Do Nash values have value?” Hydrol. Process. 21 (15): 2075–2080. https://doi.org/10.1002/hyp.6825.
Seibert, J. 2003. “Reliability of model predictions outside calibration conditionspaper presented at the Nordic Hydrological Conference (Røros, Norway 4-7 August 2002).” Hydrol. Res. 34 (5): 477–492. https://doi.org/10.2166/nh.2003.0019.
Seibert, J., M. J. Vis, E. Lewis, and H. van Meerveld. 2018. “Upper and lower benchmarks in hydrological modelling.” Hydrol. Process. 32 (8): 1120–1125. https://doi.org/10.1002/hyp.11476.
Selle, B., and M. Hannah. 2010. “A bootstrap approach to assess parameter uncertainty in simple catchment models.” Environ. Modell. Software 25 (8): 919–926. https://doi.org/10.1016/j.envsoft.2010.03.005.
Shen, Z., L. Chen, and T. Chen. 2012. “Analysis of parameter uncertainty in hydrological and sediment modeling using GLUE method: A case study of SWAT model applied to Three Gorges Reservoir Region, China.” Hydrol. Earth Syst. Sci. 16 (1): 121–132. https://doi.org/10.5194/hess-16-121-2012.
Tanguy, M., H. Dixon, I. Prosdocimi, D. G. Morris, and V. D. J. Keller. 2014. “Gridded estimates of daily and monthly areal rainfall for the United Kingdom (1890–2012) [CEH-GEAR].” Accessed February 27, 2015. https://doi.org/10.5285/5dc179dc-f692-49ba-9326-a6893a503f6e.
Vogel, R. M., and A. Sankarasubramanian. 2003. “Validation of a watershed model without calibration.” Water Resour. Res. 39 (10): 1292. https://doi.org/10.1029/2002WR001940.
Vrugt, J. A., H. V. Gupta, W. Bouten, and S. Sorooshian. 2003. “A shuffled complex evolution metropolis algorithm for optimization and uncertainty assessment of hydrologic model parameters.” Water Resour. Res. 39 (8): 1201. https://doi.org/10.1029/2002WR001642.
Weglarczyk, S. 1998. “The interdependence and applicability of some statistical quality measures for hydrological models.” J. Hydrol. 206 (1–2): 98–103. https://doi.org/10.1016/S0022-1694(98)00094-8.
Xu, C. Y. 1999. “Operational testing of a water balance model for predicting climate change impacts.” Agric. For. Meteorol. 98–99 (1–4): 295–304. https://doi.org/10.1016/S0168-1923(99)00106-9.
Information & Authors
Information
Published In
Journal of Hydrologic Engineering
Volume 25 • Issue 4 • April 2020
Copyright
©2020 American Society of Civil Engineers.
History
Received: Apr 18, 2019
Accepted: Oct 4, 2019
Published online: Feb 6, 2020
Published in print: Apr 1, 2020
Discussion open until: Jul 6, 2020
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.