Concepts of Information Content and Likelihood in Parameter Calibration for Hydrological Simulation Models

This paper is intended as a polemic. Although it has been invited as a review paper, it will not be balanced in its presentation. Neither will it be particularly biased toward one or another approach to modeling parameter calibration, though clearly the authors have a history in having developed the Generalised Likelihood Uncertainty Estimation (GLUE) methodology. In fact, the thoughts in this paper are the result of a process of thinking about the so-called GLUE controversy (Beven 2006, 2008, 2012a; Beven et al. 2008, 2012a; Clark et al. 2012) over a long period of time. But the story starts a long time ago.

Once upon a time, in the days not so long ago when computers were big and slow and had little memory, there was a belief that it might be possible to find a single computer model of a catchment that was in some sense optimal. This was, in part, a response to the limitations of the computers of the time, but it was also in part a deeper form of modernist philosophy of science expressed in terms of gradually evolving toward the true model of a catchment system. This was despite the fact that many of the conceptual model structures used were very simple [indeed, this was held to be a virtue by Dawdy and O’Donnell (1965)]; despite the fact that a lot of model calibration was carried out by manual trial and error [though a number of automatic calibration techniques were being used; see, for example, Blackie and Eeles (1985)], despite the knowledge that the optimum model would depend on the objective function used, and despite the fact that the objective function response surface might be complex [again, see Blackie and Eeles (1985)], especially when it was necessary to estimate many different parameters (the original Stanford Watershed Model had on the order of 35 parameters if snowmelt was included, including multipliers for both rainfall and evapotranspiration estimates—there was a suggestion at the time that the only person who could really calibrate it successfully was Norman Crawford, who wrote the model as part of his Ph.D. thesis, supervised by Ray Linsley). In fact, some very early work on model structures was driven by a desire to produce simple response surfaces to facilitate automatic calibration (e.g., Ibbitt and O’Donnell 1974), something that has resurfaced recently in terms of using poor numerical algorithms in more complex models, resulting in greater complexity of likelihood surfaces (e.g., Kavetski and Clark 2010).

One response to these difficulties (and also the sheer number of conceptual hydrological models available) was to take an alternative view of model calibration. In this view, model parameter values would have a basis in the physics of the flow processes and would be estimated by measurement or by knowledge of the characteristics of the soils, vegetation, and channels in the catchment. This was the basis for the Freeze and Harlan (1969) blueprint for a physically based digitally simulated model of basin hydrology. Such distributed models could, at least in principle (a much used phrase in the 1970s and 1980s), have parameter values that varied from element to element in the distributed solution, with the result that thousands, rather than tens, of parameters were now required. Such models have persisted, of course, despite both the physics of their conceptualization and the possibility of estimating effective parameter values being questioned (e.g., Beven 1989, 2001a, 2002a; Grayson et al. 1992). Some distributed models, e.g., Soil and Water Assessment Tool (SWAT), even provide databases of default parameter values for different conditions and have been used worldwide on this basis. This makes it easier for users to simulate even ungauged basins—but should really not be expected to guarantee good results. The extrapolation of model parameter values from one catchment to another is fraught with difficulty, though in some cases it can work even when land use is changing (e.g., Buytaert and Beven 2009). This was one of the drivers behind the International Association of Hydrological Sciences IAHS Prediction of Ungauged Basins (PUB) initiative (e.g., Sivapalan 2003), to see whether better hydrological understanding could lead to more success in transferring parameter values to new catchments (Blöschl et al. 2013).

Another strain of research concerned itself more directly with the problem of parameter calibration on complex, high-dimensional response surfaces. Various techniques have been developed since the 1980s, including Shuffled Complex Evolution (SCE), simulated annealing, particle swarm optimization, Pareto optimization, and other algorithms. All were attempts to find a global optimum (or Pareto set of optima in the case of multiple competing objective functions) while being robust to the distractions of local optima and local discontinuities in a surface. The SCE-University of Arizona (UA) algorithm in particular, developed at the University of Arizona, was made freely available and has been very widely used in hydrological modeling, including being the basis for other more computationally intensive algorithms [e.g., Vrugt et al. (2009), Laloy and Vrugt (2012)]. It is not totally immune to the problem of local optima, however. Early on, the study of Duan et al. (1992) demonstrated how starting the algorithm from different randomly chosen points in the model space had a high probability of finding parameter sets close to the global optimum but might occasionally fail even for only a six-parameter model.

Over the same period of time, the question of uncertainty associated with the predictions of hydrological models has also assumed greater importance. This was partly a result of the general increase in computer power available to hydrological modelers (in the 1970s and 1980s, estimates of uncertainty were generally limited to analytical statistical calculations around some so-called best fit or maximum likelihood model). More computer power means that either much more complex model structures or much finer spatial discretizations can be implemented or, alternatively, that many more runs of a model can be made. One way of using those many runs is in uncertainty assessment [Monte Carlo experiments have been carried out even with the SHE model; see Vazquez et al. (2009)]. It was also partly a result of the more general recognition of the uncertainties associated with environmental models that can arise from many different sources and be manifest in complex ways (e.g., Beven 2009, 2013). Uncertainty estimation can be important in the modeling process if it makes a difference in decision making, affecting either the decision that is made (e.g., Todini 2004) or the way that a decision is made (e.g., Beven 2011). But the different sources of uncertainty are complex and involve a lack of knowledge as well as random natural variability (epistemic as well as aleatory errors). Thus there is an important issue about how far formal statistical theory can be used to represent different types and sources of uncertainty [e.g., the recent exchange of Beven et al. (2012a) and Clark et al. (2011, 2012), as well as Montanari and Koutsoyiannis, this issue).

As pointed out by the referees on this paper, all methods of uncertainty estimation can be considered to be statistical in the sense that the only evidence that is available about potential distributions of future errors are those errors that have been seen in conditioning a model or models during calibration. The present authors agree but wish to distinguish between formal statistical methods of uncertainty estimation that assume that errors are fundamentally aleatory and methods that try to take more explicit account of epistemic errors, at least in trying to reflect the expectation of reduced information content relative to purely aleatory assumptions when estimating prediction uncertainties.

Formal statistical methods involve the evaluation of a likelihood function for different model runs, which depends on assumptions made about the modeling errors (including assumptions about distributional form; structured bias, correlation, and heteroscedasticity; stationarity; and the information content of a single residual). Where the formal assumptions hold, the likelihoods and uncertainties associated with the predictions have a formal probabilistic interpretation. The question is, however, whether the relatively simple assumptions that are generally made are adequate in the face of uncertainties arising from lack of knowledge. By definition, of course, not enough is known about epistemic errors, so any uncertainty estimation method that tries to allow for epistemic error cannot ensure a formal probabilistic interpretation. It might also fail in prediction, but this will also be the case for formal methods that treat epistemic errors as if they were aleatory in calibration when the epistemic errors in prediction are (at least for some events) rather different. This argument hinges on how far error distributions can be considered stationary in going from calibration to prediction periods. The question of stationarity is complex and considered further in what follows.

In fact, the main questions that arise in calibrating hydrological simulation models have not changed in the last 30 years or more (e.g., Nash and Sutcliffe 1970; Kirkby 1975; Gupta and Sorooshian 1985; Gupta et al. 1998, 2009; Kuczera and Mroczkowski 1998; Beran 1999; Vrugt et al. 2002; Liu and Gupta 2007). They can be summarized as follows:

It is these general questions that this paper aims to address from a hydrological (rather than information theory or statistical) perspective. Reference will be made only to simulation models. Models used for forecasting the N-step-ahead output variables given current values of those variables require a somewhat different perspective [this includes the increasingly common activity of using off-line remote sensing data assimilation to correct the predictions of distributed simulation models; see for example Pauwels et al. (2001), Rodell et al. (2004), and Reichle et al. (2008)]. The use of data assimilation in forecasting can be useful for practical applications, but it is not simulation in the sense used here.

Simulation models are needed in practical applications of predicting the impacts of future change and in predicting the response of ungauged areas, situations where data assimilation is not possible. Simulation models are also an important means of demonstrating improvements in hydrological understanding and science (whereas data assimilation can compensate for deficiencies in either observations or science, often in very useful ways when forecasts are required in real time). The question, then, is how far scientific progress can be demonstrated given the limitations and uncertainties in the modeling process that affect the calibration of model parameters (Beven 2000, 2002b; Beven and Freer 2001; Beven and Westerberg 2011; Westerberg et al. 2011a; Beven et al. 2011; McMillan et al. 2010, 2012).

In previous work (Beven 1989, 2002b, 2006, 2012b) the concept of a perceptual model of hydrological processes and the way in which any mathematical or conceptual model of those processes necessarily involves a gross simplification of the perceptual model were discussed. A perceptual model can include all the complexities, heterogeneities, and nonstationarities in the functioning of a catchment. It can serve as a useful test for the assumptions that are used in developing a mathematical model but is not in itself a useful predictor since it is only qualitative in nature. Some processes, such as preferential flows, can be perceived as important in catchment response, but there are no agreed or satisfactory mathematical descriptions of such processes that can be used to quantify their effect (Beven and Germann 1982, 2013; Bachmair and Weiler 2011). They are thus often ignored (which in itself might be a cause of epistemic error in prediction). And, of course, there may well be important omissions from even the qualitative perceptual process model that might be important in the actual catchment response.

One could similarly outline a perceptual model of uncertainty in the hydrological modeling process from a hydrological perspective. There are many different sources of uncertainty, all of which will affect the calibration of model parameters. Since parameter identification will interact with all the sources of uncertainty (including scale and commensurability effects), one would expect that values of parameters resulting from calibration would be effective values that might be different from prior estimations of values based on direct observations or other sources.

The main sources of uncertainty that will affect model calibration (e.g., Beven 2005, 2012a; Liu and Gupta 2007) are as follows:

Note that no parameter uncertainty is included in this list because it cannot be considered independently of the listed sources in the case of calibration or conditioning (e.g., Beven 2005, 2006, 2009). A separate source of uncertainty is an absence of data for calibration, but this is a less interesting case as the uncertainty estimated then depends only on the prior assumptions about each source of uncertainty. Prior parameter estimates will also have an effect in calibration/conditioning, but it is rare in hydrological modeling when there is confidence in defining prior distributions (and covariation) for parameter values.

All of these sources of uncertainty include both variation that can be represented as aleatory natural random variability and the epistemic errors that result from a lack of knowledge (e.g., Hall 2003; Rougier 2012; Rougier and Beven 2012; Beven 2013; Sun et al. 2012; Beven and Young 2013). A much more detailed classification, including linguistic and cultural uncertainties, can also be developed (Mayo 1996; Allchin 2004), but this simple differentiation will suffice here. This difference has a much older history (e.g., Knight 1921; Keynes 1936; Popper 1957; Helton and Burmaster 1996; Howson 2003) but has been generally neglected in hydrological simulation. It is suggested here, however, that its importance should be recognized more explicitly.

The reason it is important is that in general it is not possible to represent epistemic uncertainties by a formal statistical model with identifiable parameters because epistemic uncertainties will generally result in arbitrarily nonstationary error characteristics. Comments received on the first draft of this paper demonstrate that this difference requires some further explanation because purely aleatory errors can be considered arbitrary and because many relatively simple statistical error models might produce nonstationary characteristics for short periods of data, even if their parameters are constant and the generating process stationary in the long term. In addition, the only evidence for the nature of either aleatory or epistemic error will be from the available calibration data. If there are nonstationary characteristics that lead to quite different errors in prediction, then there can be no question of a priori evidence for them. A good illustration of this is the postaudit analysis of the predictions of groundwater models in Konikow and Bredehoeft (1992), where success depended largely on how well future boundary conditions had been estimated [also a form of epistemic error problem; see Popper (1957) for a similar statement].

This is one reason why many modelers do not see why epistemic errors should not be treated in the same way as aleatory errors within a formal statistical framework (see the arguments of Montanari and Koutsoyiannis, this issue). This also has the claimed advantage of objectivity in the sense that the assumptions of an error model can be checked against the actual model residuals (it is, indeed, good practice to do so). One recent referee even went so far as to say that the distinction between aleatory and epistemic error is not important because in principle statistical error models can be “infinitely complex.” That just begs the question of how such a complex error model might be identified for arbitrary epistemic errors [and how that error model might interact with the identification of the underlying hydrological model; see, for example, the discussion of Beven (2009)].

There is that word arbitrary again. So how is arbitrary different from aleatory, and why is the difference important? This is really a question of time scale of variability relative to the time scales in calibration and prediction, and particularly in calibration, where all the information content lies that informs the model predictions. From the law of large numbers under the assumptions of an aleatory statistical error model (after allowing for bias or autocorrelaton, for example), it is expected that the low-order moments of a distribution of residuals will converge quite rapidly relative to having a reasonably long length of available calibration data. Indeed, it is hoped that the calibration period will be long enough to ensure convergence to the moments of the underlying population so as to increase confidence in the error model parameters when they are used to estimate uncertainty in prediction. The Gaussian model of likelihood, under the aleatory assumption, also reflects this. The dominant term is normally proportional to ${({\sigma }_{\epsilon }^2)}^{-N/2}$, where ${\sigma }_{\epsilon }^2$ is the variance of the model residuals and $N$ is the number of residuals. Therefore, when $N$ is very large (as is quite common in the calibration of hydrological models), two models with very similar error variances will have orders-of-magnitude differences in likelihood. In effect, the likelihood space is stretched hugely by the aleatory error assumption.

That might, of course, be absolutely the right thing to do if the errors really are aleatory (though it still seems a little odd given the possibility of model structural error that might itself lead to nonstationarity of residual characteristics). But what if the errors are arbitrary in the sense of having odd periods of rather different characteristics during the calibration period in any one of the different sources of uncertainty in the modeling processes. Most modelers will recognize this situation. Even models that fit most of the data well will have a poor fit to some odd events [e.g., comment of Beven (2009) on the Leaf River data used in Vrugt et al. (2009)]. In this case there may be no asymptotic distribution for any of the sources of error for the time scale of an application, implying also that the arbitrary errors in prediction might be different from those in calibration. The point is that treating such errors as if they were aleatory will overestimate the information content of the calibration period residuals.

It is not difficult to identify potential errors of this (epistemic) type. Consider the very practical example of a perceptual model of the uncertainties associated with rainfall inputs to a hydrological simulation model. The uncertainty will be different for different numbers of rain gauges. It will be different depending on rain gauge design, placement, and exposure (particularly in catchments with a significant elevation range and wind effects on catch). It will be different depending on whether rain gauge information can be combined with radar data (with all of the epistemic uncertainties involved in converting the radar return signal into rainfall intensities). One might have the perception that for a given number of rain gauges the epistemic error might be different for small patchy rainstorms than for synoptic events, but even in synoptic events there might be cells moving over a catchment that are poorly represented by a sparse measurement network. Most of these sources of uncertainty are epistemic, not aleatory. Their characteristics might be quite different from event to event and in some cases might lead to observations that are hydrologically inconsistent, such as those events where apparent runoff coefficients for an event are greater than 1. Such events should not be used in model calibration (Beven and Westerberg 2011; Beven et al. 2011, see subsequent discussion), but this has rarely been considered in calibration exercises (partly because of the difficulty of estimating runoff coefficients or identifying inconsistent data for individual events). In fact, hydrological reasoning also suggests that epistemic errors of this type can have an impact beyond the single event because of the way input errors are processed through the dynamics of a model to produce an effect on the antecedent state of the (modeled) catchment in predicting the next event.

Similar perceptual arguments can be made about other boundary conditions required to run a model and about the observed discharges used to evaluate model predictions in calibration [since discharge is generally a constructed variable dependent itself on a model; see Beven et al. (2012b) and Westerberg et al. (2011a)]. This is also not something new: in their seminal paper Stephenson and Freeze (1974) pointed out that the validation of a distributed hillslope model would be impossible because it was not possible to know both the initial and boundary conditions for the simulation of any real hillslope with sufficient accuracy and precision. One would also expect the nature of errors to be nonstationary for good hydrological reasons. The way in which boundary condition uncertainties interact with model structural uncertainties might be quite different in predictions of the rising limb of a hydrograph compared to the falling limb of a hydrograph. One would also expect that the magnitude of prediction uncertainties might increase with predicted discharge and that any time step correlation in model errors might be quite different for the rising and falling limbs. Such effects should be expected—and they should be expected not to have purely random characteristics.

Thus, given the potential for both aleatory and epistemic sources of uncertainty in hydrological simulation, how should one go about assessing the information content of a set of data that might be used in parameter calibration? Traditionally, information theory has assessed information content in terms of entropy measures [e.g., going back at least as far as Amorocho and Espildora (1973); see also Singh (1998) and Weijs et al. (2010)], while systems analysis and statistics have made use of Akaike, Bayes, Deviance, Young, and other information criteria that focus on the assessment of model fit and parameter uncertainties. Both approaches have been applied widely in hydrological modeling but seem to represent the wrong place to start from a hydrological perspective. Entropy measures [and the equivalent U-uncertainty measures in a possibilistic framework; see Beven and Binley (1992)] are based only on the shapes of distributions, so that any information about error sequences is largely lost (except when looking at lagged cross-entropy measures). Statistical information measures, on the other hand, assume that all errors can be treated as if they were aleatory, which is also clearly not the case. A new approach seems necessary.

Starting from a hydrological perspective, such an approach should be event based. Immediately the practical import of the nature of uncertainties in the modeling process becomes a little clearer. Let us begin with an assumption that all events, a priori, should contribute equally to the information content of a calibration data set. It is known, however, that such an assumption is certainly not appropriate: the real contribution of information for different events will depend on multiple factors as follows:

Ideally, of course, one would wish to quantify the information contributed by each event, but nearly all of these factors are subject to epistemic uncertainties. For example, unusualness would be an advantage if the data are consistent. If a model can then reproduce the catchment response of an unusual event, then one’s confidence in that model as a simulator would increase [point c above; see also Singh and Bárdossy (2012)]. But an event might be unusual only because of epistemic uncertainty in the available observations (a local convective cell of rainfall happens to be centered on a rain gauge such that the estimate of input rainfall over the catchment is unusually high, but the discharge hydrograph shows little or no response). In some cases, such inconsistencies might be easily recognizable [e.g., when runoff coefficients are greater than 1 or exceptionally low; see Beven (2009), Beven et al. (2011), and the example in what follows]. Any hydrologist with modeling experience will be able to recognize a variety of other similar cases. Such events might be disinformative in the sense that the data might be misleading in the specific context of the evaluation of sets of model parameter values to determine whether they will be useful in prediction. Clearly such data might be informative in other contexts (e.g., in providing information about epistemic data errors). In many cases, distinguishing between usefully unusual and inconsistent events might be rather difficult, even more so when the inconsistency is generated by a prior event (point g above). What is clear, however, is that the effects are certainly not aleatory in nature in the sense of being drawn from a consistent and stationary statistical distribution of potential errors.

These arguments suggest that an attempt should be made to identify the relative information content of a series of events independent of any model run. Otherwise, there is the possibility of a reductio ad absurdum of declaring as disinformative all events that cannot be fitted by a particular model. Making information content conditional on the assumption that the model is a correct representation of the system seems to be an unsuitable strategy in a hydrological context (albeit traditional in both frequentist and Bayesian statistical inference, possibly after adding a model discrepancy function).

So what is the alternative? How can an evaluation of the information content (or potential disinformation) of an event be assessed for the purposes of parameter estimation? One suggestion, given enough calibration events, is to take advantage of the tension between the consistency and information redundancy of event data. There is a (perceptual) expectation that events with similar antecedent conditions and similar rainfall amounts and intensities will have similar observed responses. Thus, classifying events according to antecedent conditions and rainfall characteristics (based on the data alone) will result in a division of events into different clusters, within which the events should be more or less consistent with each other. Unusual events in each cluster would be expected to stand out and could be assessed as to whether or not they might be informative for parameter calibration purposes. Multiple events that are similarly unusual might indicate the need for a new class of events.

In this way, it is possible to assess a likelihood weight associated with each storm period that takes into account the characteristics listed earlier that should either increase or decrease the potential information content of that period of data. This approach is explored in the example application in what follows, where clustering based on storm characteristics is used both in the determination of disinformative periods and in the formulation of a measure of storm information content.

To illustrate the practical application of the preceding discussion, consider the calibration of a rainfall-runoff model to observed data. The observed data used are discharges and hourly rainfall totals for the $322{\mathrm{km}}^2$ headwater catchment of the South Tyne (U.K.) above Featherstone. This upland catchment is predominantly covered by moorland that is used for rough grazing and overlies a Carboniferous sequence of coal measures, sandstones, mudstones, and shales. Calibration was performed using data from 1990 to 1997, with the data from 1998 to 2003 held back for evaluation.

The discharge data $y_t$ (indexed by time) are observed using a compound Crump weir. The weir contains the flow at all recorded stages and remains modular throughout its range. The discharge series is generally considered to be of good quality and is considered to represent the so-called natural flow to within 10% at the 95th percentile flow.

There are five recording rain gauges within the catchment. These generally recorded higher rainfall totals when compared with storage gauges within the catchment. Analysis of the rainfall data and the spatial distribution of the gauges indicate there is scope for both the under- and overestimation of the inputs to the catchment under different rainfall patterns. In the modeling exercise that follows, catchment average rainfall estimates, $r_t$, are computed from the recording rain gauges using a Thiessen polygon method.

The catchment averaged rainfall, along with a synthetic potential evapotranspiration series, $p_t$, based on annual and daily sinusoids, is used to force the hydrological model that is to be calibrated. The model is a lumped storage model consisting of a single tank. The volume of water stored in the tank, which has a maximum capacity of $s_{\max }$, is denoted by $s_t$ and evolves according to Eqs. (1)–(3):

The effective rainfall is given by $u_{t+1}=\gamma s_t+\max (s^{\prime }-s_{t+1},0)$. This is routed through two parallel tanks expressed as a linear transfer function [Eq. (4)] using the backward shift operator $z^{-1}$ (i.e., $z^{-i}{w}_t=w_{t-i}$) [see, for example, Young and Beven (1994), Young et al. (2004)]. The output of these tanks is then summed to give the model output $x_t$. Limiting the parameters such that $0<{\alpha }_1<{\alpha }_2<1$ and $0<\rho <1$ ensures that the linear routing is mass conservative, with $\rho$ being the fraction of the effective input going to the tank with the faster response

The model has five parameters $\theta =(s_{\max },\gamma ,{\alpha }_1,{\alpha }_2,\rho )$ that require calibration.

As Beven (2008) observed, there remains a great deal of uncertainty about uncertainty estimation in hydrological modeling. In this paper an attempt has been made to analyze how this is the result of the epistemic uncertainties inherent in the hydrological modeling process and to propose some ideas about how to deal with assessing the relative information content of hydrological data (independently of any model) and how it might influence model conditioning based on hydrological reasoning. As noted earlier, by the very nature of epistemic errors, there can be no right answer without the additional information that would be necessary to define their characteristics. The proposed method consists of the following steps within a GLUE framework that have been demonstrated in the simple application presented:

Software supporting these steps for the rainfall-runoff modeling case will be available from the authors.

Hydrology remains a subject limited by its measurement techniques. As was pointed out previously (e.g., Beven 2001b), even the water balance equation for a catchment cannot be verified by measurement without allowing for significant uncertainties in each of the input, output, and storage terms. It has been suggested here that the fact that these uncertainties involve epistemic and aleatory characteristics has important consequences for model calibration, hypothesis testing, and prediction. For individual events input and output data can be shown to be hydrologically inconsistent and may be disinformative to the model calibration and testing processes. It does not appear that the issue of epistemic error in the hydrological data will go away for the foreseeable future, but a proper evaluation of informative data is probably a prerequisite for any real improvements in testing hydrological model structures as hypotheses about catchment response (Beven 2010, 2012a; Beven et al. 2012a; Clark et al. 2011, 2012).

There has been long experience in hydrology with the performance of hydrological models in prediction or validation being worse than in calibration (though this is not always the case, a particular realization of errors can give better performance in prediction). Clearly one explanation for this is the possibility of epistemic errors that are quite different in calibration from those in prediction. Removing inconsistent or disinformative events from the calibration process might lead to more robust predictions of hydrologically consistent events in validation but, as Fig. 6 shows, does not guard against the failure to predict inconsistent events. The possibility of defining additional prediction bounds for these types of events needs to be explored in future work. This may be one way of separating robust model calibration and testing from more subjective decisions about projections of potential epistemic errors.

KB recently rediscovered that equifinality was first used in the context of hydrological model calibration in his Ph.D. thesis in 1975, but the 1993 prophecy paper is the first reference in print. To his knowledge, the first use of equifinality in this sense is in the work on general systems theory by Ludwig von Bertalanffy (1950, 1968). Culling (1957) and later Chorley (1962) first used it in relation to the development of similar landforms from different initial conditions by different histories, though Culling (1987) later preferred an interpretation in terms of nonlinear dynamics. A fuller discussion in this context is given in Beven (1996).

KB started doing work on Monte Carlo simulations of hydrological models while working at the University of Virginia in 1980 on a CDC6600 mainframe computer. Discussions with George Hornberger at that time helped shaped the ideas that led to the GLUE methodology, published with Andy Binley in 1992. Since that time, discussions with many others have helped to clarify the issues of aleatory and epistemic uncertainties and their impact on likelihoods. This work is a contribution to the CREDIBLE consortium funded by the U.K. Natural Environment Research Council (Grant NE/J017299/1). Comments from Bellie Sivakumar, Alberto Montanari, and an anonymous referee helped to improve the paper.