The Starbucks store near our office offers beverages in three sizes: tall, grande, or venti (or small, large, and very large, for the uninitiated). What’s the best choice? The venti is largest, and it surely includes that extra java jolt that you need the morning after a late night in front of your workstation. That’s a jolt that the diminutive tall could never deliver.
But I’ve noticed that when I purchase the venti size, a portion of the beverage often sits in my cup until it is cold, ending up in the sink or in the sad potted plant in my office. Venti really is more than I needed. Recently, I’ve wised up and scaled back, getting the grande instead. It is just barely enough: it meets my need with no waste.
We should use similar logic when choosing models that we select or create to provide information for water resources planning and management. We don’t always need the venti model—a model that is big, all-encompassing, complex, or cutting edge. Instead, we need models that are—to borrow a term from current software development jargon—just barely good enough.
What Is “Good Enough”?
Hold on a minute. Don’t get too excited by the notion of doing engineering work that is just barely good enough. I am not advocating the use of models that are inaccurate, incomplete, poorly done, or just plain wrong. Those are not good enough. Instead, I’m promoting using models that meet the project requirements, nothing more and nothing less.
This selection strategy is not new to us in water resources planning. We use it as the standard for selecting from among alternative flood-defense project sizes, for example. Consider the simplified case for which benefit and cost are a function of capacity as illustrated in Fig. 1. Cost is the dashed line, and benefit is the solid one.
Fig. 1. Optimal size is just barely good enough
Clearly, a bigger project yields greater benefit in this example. But selecting the biggest (venti) alternative is not always the best strategy, for it fails to consider the trade-off with cost. In the illustration, larger sizes have cost that exceeds the benefit. The same may be true of models.
Our standard is to consider the net: the difference between benefit and cost. The optimal choice—the choice that is just good enough—is the capacity that yields the maximum net benefit. In the figure, the optimal choice is denoted with a dotted vertical line. A project that is larger, to the right of the line, yields greater benefit, but it costs more than it returns in benefit. A project that is smaller, to the left of the line, could be made more cost-effective by an increase in capacity. But you probably understand that already.
Our Current Decision Making Standard
Even though we understand well the net benefit logic and apply the rules for selecting good enough projects for flood defense, we—and I include myself in this finger pointing—haven’t done as well applying the same logic in making decisions about models. Instead of selecting good enough models, we succumb to the temptation to select the venti model.
Consider, for example, the task of selecting flow routing models. Advances in open channel flow modeling software make it easy to develop unsteady flow models of complex networks. If coupled with advanced terrain data-processing features of modern GIS tools, the models can produce detailed, accurate floodplain maps for land-use regulation. The continued investment by FEMA in such technology is evidence that the damage reduction resulting from decision making with the improved maps offsets the cost of using the advancing technology to produce them.
Now that such models are readily available, are they the optimal choice for all flow routing problems? For example, should we use the complex models and GIS mapping extensions in operational settings to route reservoir releases to forecast downstream stages? Or is it acceptable to use simpler hydrologic routing models coupled with reliable ratings? Are they good enough?
We can answer that question correctly only after considering the benefits and costs of the alternatives and the characteristics of the channels modeled. We must consider that in operational settings, we cannot fail to make a forecast—one that is sufficiently accurate and issued early enough to permit actions that protect lives and property. Models that do not converge or that are unstable are not acceptable in that context. Thus, if we use an unsteady-flow model, it must be fine-tuned to ensure that the “hiccups” that may occur when the models are used for floodplain mapping do not occur during real-time forecasting. This fine-tuning incurs a cost, a cost that may not be justified if the simplified routing models yield forecasts that are within acceptable tolerances. By using overly complex models, we may err on the venti side of grande.
However, we can err on the tall side of grande. For example, if elevations in the channel are influenced by tides downstream, a simple hydrologic routing model is not appropriate: it is not good enough. The additional cost of developing and using the unsteady flow model is justified in that case. In fact, it is required.
Considerations in Finding “Good Enough”
Grayson and Chiew (1994) suggest that to find models that are good enough (my words, not theirs), we must consider the following:
•
Objectives of the overall exercise. Is the model to be used by researchers for fundamental research; by modelers or technical experts to give final answers; by managers to evaluate the effects of management options; or by the community to assist in education, technology transfer, and ownership of solutions? Only if we are clear on the reason for using the model can we select a model that meets the requirements of its intended use, is sufficiently accurate, is sufficiently detailed, and is flexible enough to change as our concept of good enough evolves.
•
Access to expertise for developing and using the model. Grayson and Chiew note that models don’t generate knowledge—they simply present what we already know. Models that are good enough present that information in a manner that is usable by the intended audience, without going beyond that. Models developed for use by modelers or technical experts (geeks) may be structured to reveal insights at lower cost than models properly developed for casual users or managers. Technical experts are satisfied with tables of numbers, whereas managers may want color-coded maps and charts. If we do not have ready access to the experts to interpret the results, good enough may require a more sophisticated interface.
•
Data availability. Availability of data is often the binding constraint in model selection. As Grayson and Chiew (1994) point out, it is possible to simulate virtually anything with the computer. But whether the results bear any relationship to reality depends on the data available for calibration and application. We could, for example, select a watershed model with fine spatial resolution of infiltration and overland flow. However, if we have no streamflow data within the watershed to permit calibration or even verification of that model, the model is not good enough. The cost of developing the higher-resolution model is not offset by the benefit—at least not by a tangible benefit that we can demonstrate clearly.
•
Time and money resources. A good enough model will be developed and applied within the limits of the time and money available. If a model produces great results but provides them too late to have an effect on the decision making, it is not good enough. Similarly, if the model selected and used was the one in hand when the money ran out, chances are that it is not good enough.
Tips for Finding “Good Enough”
Here, drawn from software engineering (Bach 1997; Ambler 2004, 2005; Hill 2005; Bach 2003) and hard knocks, are some suggestions about finding good enough when making modeling decisions:
1.
Develop a big picture first. We should develop a vision for and design of a complete decision support system (DSS) rather than focus narrowly on smaller aspects of the system. Don’t worry—we can still dive into the details when appropriate. But having the “vision thing” will help identify the expertise needed, the data required, and the time and money required.
2.
Bring broad expertise to the task. The requirements for most modern DSSs in water resources planning and management are complex, since the DSSs deal with a wide variety of economic, environmental, legal, and technical issues. To be good enough, the models must address all the relevant issues. Modular design helps.
3.
Identify the users, engage them, and work with them to set priorities. We must rely on the users of our DSSs to identify requirements, and we must select the models to meet those requirements (but not more). And of course, when time and money are limited, we must rely on users to prioritize their requirements. Doing so enables us to work on the most important tasks, finding the models and creating a DSS that is good enough. The broad vision and broad expertise are therefore especially important. If we must leave something out now, at least we will know where it will fit—and how—if the resources are available later.
4.
Don’t get hung up with technology dependency, unless doing so is necessary. A successful system is not limited to one that uses only one vendor’s technology. We should focus on the needs, not on the technology that meets the needs.
5.
Design for changing requirements. Change happens. Information users often don’t know what they want or need. If they do know, they may not know how to communicate that need, other than to tell us IKIWISI (I’ll know it when I see it) or IKIWINI (I’ll know it when I need it). And we must recognize that the external environment changes. New demands for information may arise, new information technology may become available, new water users may come and old ones may go, and new modeling capabilities may be presented.
6.
Design or select models that require only data that are available. Woolsey and Swanson (1975) make strong and witty arguments that, with many models, “… the data is not present, or if it is present, it is not in the right form.” To avoid this outcome, we must identify the data available and include, in our assessment of good enough, the cost to acquire more or different data to feed the algorithms.
7.
Display proposed system features. Proposed plans and designs for a DSS are important communication channels. We should make plans accessible. We know of a case in which the user of a DSS threatened to “jump on the chest” of the system developers if information was presented as a proposed table. Instead, he wanted to see a map. Here, nothing short of those GIS extensions would be good enough.
Conclusion
Just as we do with coffee and water-resources investment planning, we must consider the trade-offs of benefits and costs in selecting models. Increased complexity commonly comes with increased cost. Increased complexity, however, is not necessarily accompanied by increased benefit in insights gained by using the models.
We must consider the objectives of the overall exercise, our access to expertise, the data available, and our time and money resources while we make decisions about models. If grande is good enough, we should select it. And, as one reader of a draft of this editorial pointed out, we can even request and get a short coffee (or a short model)—even though it is not on the vendor’s menu—if it is good enough to meet our needs.
Grayson, R. B, and Chiew, F. H. S. (1994). “An approach to model selection.” Hydrology and Water Resources Symp.Institute of Engineers, Adelaide, Australia.
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
0733-9496(2006)132:1(1)/asset/1cb63ca3-a894-4951-a9dc-bb0a75073a9c/assets/images/1.jpg)
