Incorporating Uncertainty and Variability in Engineering Analysis

Though there is some ambiguity as to the exact meaning and difference between variability and uncertainty, the following definitions from the book Risk Analysis—A Quantitative Guide by David Vose (2000) provide a good distinction between these terms. “Variability is the effect of chance and is a function of the system. Uncertainty is the assessor’s lack of knowledge (level of ignorance) about the parameters that characterize the physical system that is being modeled.” Vose then goes on to say that “total uncertainty is the combination of uncertainty and variability” and that “these two components act together to erode our ability to be able to predict what the future holds.”

There are many cases where the public is being exposed to the concepts of variability and uncertainty. Weather forecasts no longer state that it will rain tomorrow. Rather, they are posed in probabilistic terms: “there is a 90% chance of precipitation.” This statement actually includes both concepts of variability and uncertainty. Variability recognizes that it may rain in one neighborhood while another area a few miles away may not receive rain. There is also uncertainty in such a situation because the mathematical models used to predict weather are not perfect. Another example in the area of weather is in the prediction of hurricane tracks. Instead of offering one probable track, modern hurricane predictions provide a cone of uncertainty with a central line representing the most likely movement of the hurricane. The outer limits of the cone represent the spatial area within which meteorologists have determined, to some level of confidence, that the hurricane will likely strike. Another very different example is the way in which polling results for elections are reported. Candidate A is reported to be favored by 48% of the voters, while candidate B is supported by 45% of the voters with a margin of error of plus or minus 4%. This statement actually means that the pollsters think that candidate A is slightly ahead, but due to the underlying variability and uncertainty, the race is still too close to call. Variability in this case may be due to the fact that the interviewees may not actually know whether they will vote and for whom they will vote, while uncertainty is caused by the size and makeup of the poll and the polling techniques.

In the field of water resources engineering there are numerous examples of variability and uncertainty. In some studies, variability and uncertainty are explicitly considered, in other cases they are more subtlety incorporated into the analysis, while in all too many cases they are completely ignored. Hydrologic analysis generally includes an explicit representation of variability/uncertainty. For example, floods and rainfall are typically treated in a probabilistic fashion when designing dams, levees, and other structures by specifying probability distribution functions, rather than point estimates.

However, in many areas of water resources planning, variability and uncertainty are largely ignored. Water supply planning projects typically express system performance in terms of “safe” or “firm” yield. Recent droughts have demonstrated that there is no such dependable level. In most water system master plans, water consumption is typically projected for the next 5, 10, and 20 years. However, the degree of uncertainty in these predictions are not generally stated and the implications of falsely precise predictions on resulting designs are not provided. Similarly, uncertainty in parameters used in distribution system models is usually ignored rather, a most likely parameter value is selected and used in the analysis. Walski (2000) very aptly addressed this issue in his paper “Model Calibration Data: The Good, the Bad and the Useless.” He shows that errors in field data can be quantified and that, for example, uncertainty in measuring flow and head losses during a $C$ -factor test can render the resulting estimates essentially useless.

Water distribution systems analysis is particularly challenging in terms of the examination of uncertainty due to the spatial and temporal aspects of the data involved, the use of long-range forecasts, temporal and spatial autocorrelation, the influence of significantly different time and space scales that must simultaneously be included in the analysis (forecast in years, dynamic behavior in minutes), as well as the high degree of “model uncertainty” in terms of understanding all of the operative mechanisms (for example chlorine residual decay, or THM growth) in a manner sufficient for prediction. Further, it is very difficult to define an aggregate measure of water distribution system performance that takes into account temporal and spatial factors, and both hydraulic and quality aspects. Analysis is typically performed such that the system reliably performs above certain standards at the lowest cost, but metrics that reward increased performance, or allow for occasional substandard performance, are not widely used.

Another critical but little-noted issue is how best to make choices when presented with information about uncertainty. Most decision makers (and most engineers) prefer a clear set of choices, typically based on some single aggregate average measure of performance, and most analyses provide no more than that. When multiple measures of performance are incorporated, the problem becomes a multicriteria decision issue, adding additional complexity to the choice problem. When uncertainty is included, choices must frequently be made between better “average” performance and reduced risk of poor performance. The well-known analysis technique of Monte Carlo simulation (to be discussed), can provide probability distributions of performance measures rather than simple point estimates. From these distributions, simpler statistical measures, such as a mean and standard deviation, can be developed, but the decision maker must still make trade-offs in their final decisions. Trade-off examples include the explicit choice of a higher mean performance or a reduced risk (this problem is classic in financial investment decisions such as stock market portfolio analysis, where the “risk-return” issue is paramount [Males 2002]) or trade-offs between lower costs and increased reliability.

Thus, not only is it necessary to display the range of choices, with information as to the uncertainty surrounding each choice, it is also necessary to provide paradigms for decision making when uncertainty is present and explicitly stated.

There are several quantitative methods for incorporating uncertainty and variability in an analysis. The most commonly used approach and one that is most accessible to researchers and practicing engineers is Monte Carlo simulation. This approach, named after the gambling capital of Monaco, uses the concept of “rolling dice” as part of a simulation process. A deterministic model of a process is developed and applied for many iterations with the values for model parameters for each iteration randomly drawn from probability distributions rather than just choosing the best estimate for the parameter. The results of the analysis can then be expressed probabilistically. Monte Carlo analysis is especially attractive because of the relative ease in applying it. There are several “add-ins” available for spreadsheet software, or computer programming can be used to develop custom solutions.

Proper application of Monte Carlo simulation requires that:

The discipline of working through a problem in this framework is extremely valuable to the practitioner, forcing a deeper understanding of the problem, and an explicit examination and admission of the uncertainties that are present in the analysis. Though widely used in many areas of analysis for almost a half century, there are relatively few examples of application of Monte Carlo simulation in water distribution system analysis. Murray et al. (2004) are applying Monte Carlo simulation to study the vulnerability of water distribution systems to terrorist contamination threats based on a conceptual model developed by Propato and Uber (2004). Variables such as location and the characteristics of the contaminant are treated probabilistically through simulation of a large number of scenarios. Since Monte Carlo simulation requires examination of many iterations, running a model of a large system repeatedly requires extensive computer resources. In this study, the problem is being solved by using a large parallel processing capability to simultaneously simulate many scenarios. Similarly, in a paper presented in this special issue of the Journal, Nilsson et al. (2005) use Monte Carlo computer experiments to simulate a deliberate biochemical assault on a municipal drinking-water distribution system in order to generate empirical frequency histograms of mass-dose loadings at select nodes in a typical water distribution system.

Harding and Grayman (2002) applied Monte Carlo simulation in an historical reconstruction of distribution system contamination as part of a legal case in California. Variables such as wellhead concentrations, system operation, and demands were treated probabilistically in the analysis. Rather than just stating a single value, analysis results were presented in probabilistic terms, expressed, for example, as 95% confident that the true value of the concentration of the contaminant of concern fell between an upper and lower value. In a recent federal case, a similar technique was applied in Hawaii (Linda Akee et al. vs. the Dole Corporation et al. 2004). The significance of this case was that the methodology incorporating Monte Carlo simulation in water distribution system analysis passed the rigid “Daubert test” under the Federal Rules of Evidence, thus legitimizing this procedure in the eyes of the court.

In order to effectively consider uncertainty and variability in water-resources decision analysis there is both a need for increased sensitivity to the issue and improved training in this area. Most students are exposed to some basic concepts of probability and statistics. In addition, some universities offer either required or optional courses that specifically address uncertainty.

However, in order to effectively incorporate uncertainty into engineering analysis, both engineering students and practicing engineers must be further sensitized into the pervasiveness of this concept. In reality, every piece of information and every result of a model application should be viewed in a multidimensional framework. The narrow view is that this information or model result is the “correct” answer. The broader view considers each piece of data or model result as one observation drawn from a probabilistic distribution. That observation may be considered to be the most likely value, but there are other possible values with some finite level of probability that must also be considered in any analysis. Once this concept is accepted and used, then the engineer becomes more sensitive to the implications and impacts that may result due to uncertainty.