Abstract
Water systems planners have long recognized the need for robust solutions capable of withstanding deviations from the conditions for which they were designed. Robustness analyses have shifted from expected utility to exploratory bottom-up approaches which identify vulnerable scenarios prior to assigning likelihoods. Examples include Robust Decision Making (RDM), Decision Scaling, Info-Gap, and Many-Objective Robust Decision Making (MORDM). We propose a taxonomy of robustness frameworks to compare and contrast these approaches based on their methods of (1) alternative generation, (2) sampling of states of the world, (3) quantification of robustness measures, and (4) sensitivity analysis to identify important uncertainties. Building from the proposed taxonomy, we use a regional urban water supply case study in the Research Triangle region of North Carolina to illustrate the decision-relevant consequences that emerge from each of these choices. Results indicate that the methodological choices in the taxonomy lead to the selection of substantially different planning alternatives, underscoring the importance of an informed definition of robustness. Moreover, the results show that some commonly employed methodological choices and definitions of robustness can have undesired consequences when ranking decision alternatives. For the demonstrated test case, recommendations for overcoming these issues include: (1) decision alternatives should be searched rather than prespecified, (2) dominant uncertainties should be discovered through sensitivity analysis rather than assumed, and (3) a carefully elicited multivariate satisficing measure of robustness allows stakeholders to achieve their problem-specific performance requirements. This work emphasizes the importance of an informed problem formulation for systems facing challenging performance tradeoffs and provides a common vocabulary to link the robustness frameworks widely used in the field of water systems planning.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
Portions of this work were funded under grants from the Sectoral Applications Research Program (SARP) of the United States National Oceanic and Atmospheric Administration (NOAA) climate program office (Award No. NA110AS2310144), the National Science Foundation through the Network for Sustainable Climate Risk Management (SCRiM) (NSF cooperative agreement GEO-1240507), and the National Institute of Food and Agriculture, U.S. Department of Agriculture (WSC Agreement No. 2014-67003-22076). The views expressed in this work represent those of the authors and do not necessarily reflect the views or policies of NOAA, NSF, or the USDA.
References
Balling, R. J., and Richard, J. (2000). “Pareto sets in decision-based design.” J. Eng. Val. Cost Anal., 3(2), 189–198.
Bankes, S. C. (2002). “Tools and techniques for developing policies for complex and uncertain systems.” Proc. Nat. Acad. Sci. United States Am., 99(Suppl 3), 7263–7266.
Bankes, S. C., Lempert, R. J., and Popper, S. W. (2001). “Computer-assisted reasoning.” Comput. Sci. Eng., 3(2), 71–77.
Banzhaf, H. S. (2009). “Objective or multi-objective? Two historically competing visions for benefit-cost analysis.” Land Econ., 85(1), 3–23.
Baroni, G., and Tarantola, S. (2014). “A general probabilistic framework for uncertainty and global sensitivity analysis of deterministic models: A hydrological case study.” Environ. Modell. Softw., 51, 26–34.
Ben-Haim, Y. (2004). “Uncertainty, probability and information-gaps.” Reliab. Eng. Syst. Saf., 85(1), 249–266.
Ben-Haim, Y. (2012). “Why risk analysis is difficult, and some thoughts on how to proceed.” Risk Anal., 32(10), 1638–1646.
Beyer, H.-G., and Sendhoff, B. (2007). “Robust optimization—A comprehensive survey.” Comput. Meth. Appl. Mech. Eng., 196(33–34), 3190–3218.
Bond, S. D., Carlson, K. A., and Keeney, R. L. (2008). “Generating objectives: Can decision makers articulate what they want?.” Manage. Sci., 54(1), 56–70.
Bond, S. D., Carlson, K. A., and Keeney, R. L. (2010). “Improving the generation of decision objectives.” Decis. Anal., 7(3), 238–255.
Bonder, S. (1979). “Changing the future of operations research.” Oper. Res., 27(2), 209–224.
Borgonovo, E., and Peccati, L. (2008). “Sensitivity analysis in decision making: A consistent approach.” Advances in decision making under risk and uncertainty, Springer, Berlin, Heidelberg, 65–89.
Brown, C. (2010). “Decision-scaling for robust planning and policy under climate uncertainty.”, Washington, DC.
Brown, C., Ghile, Y., Laverty, M., and Li, K. (2012). “Decision scaling: Linking bottom-up vulnerability analysis with climate projections in the water sector.” Water Resour. Res., 48(9), W09537.
Brown, C., and Wilby, R. L. (2012). “An alternate approach to assessing climate risks.” Trans. Am. Geophys. Union, 93(41), 401–402.
Bryant, B. P. (2009). sdtoolkit: Scenario discovery tools to support robust decision making, R package version 2.31, 〈http://CRAN.R-project.org/package=sdtoolkit〉 (Mar. 2013).
Bryant, B. P., and Lempert, R. J. (2010). “Thinking inside the box: A participatory, computer-assisted approach to scenario discovery.” Technol. Forecasting Social Change, 77(1), 34–49.
Chamberlin, T. C. (1890). “The method of multiple working hypotheses.” Science, 15(366), 92–96.
Clímaco, J. C. (2004). “A critical reflection on optimal decision.” Eur. J. Oper. Res., 153(2), 506–516.
Cohon, J., and Marks, D. (1973). “Multiobjective screening models and water resource investment.” Water Resour. Res., 9(4), 826–836.
Cohon, J., and Marks, D. (1975). “A review and evaluation of multiobjective programming techniques.” Water Resour. Res., 11(2), 208–220.
Crossley, W. A., Cook, A. M., and Fanjoy, D. W. (1999). “Using the two-branch tournament genetic algorithm for multiobjective design.” AIAA J., 37(2), 261–267.
Deb, K., and Gupta, H. (2006). “Introducing robustness in multi-objective optimization.” Evol. Comput., 14(4), 463–494.
Dessai, S., Browne, A., and Harou, J. J. (2013). “Introduction to the special issue on adaptation and resilience of water systems to an uncertain changing climate.” Water Resour. Manage., 27(4), 943–948.
Dessai, S., and Hulme, M. (2007). “Assessing the robustness of adaptation decisions to climate change uncertainties: A case study on water resources management in the east of England.” Global Environ. Change, 17(1), 59–72.
DiFrancesco, K. N., and Tullos, D. D. (2014). “Flexibility in water resources management: Review of concepts and development of assessment measures for flood management systems.” J. Am. Water Resour., 50(6), 1527–1539.
Efron, B., and Stein, C. (1981). “The jackknife estimate of variance.” Ann. Stat., 9(3), 586–596.
Frederick, K. D., and Major, D. C. (1997). “Climate change and water resources.” Clim. Change, 37(1), 7–23.
Frederick, K. D., and Schwarz, G. (1999). “Socioeconomic impacts of climate change on U.S. water supplies.” J. Am. Water Resour. Assoc., 35(6), 1563–1583.
Friedman, J. H., and Fisher, N. (1999). “Bump hunting in high-dimensional data.” Stat. Comput., 9(2), 123–143.
Gerst, M., Wang, P., and Borsuk, M. (2013). “Discovering plausible energy and economic futures under global change using multidimensional scenario discovery.” Environ. Modell. Softw., 44, 76–86.
Ghile, Y., Taner, M., Brown, C., Grijsen, J., and Talbi, A. (2014). “Bottom-up climate risk assessment of infrastructure investment in the Niger River basin.” Clim. Change, 122(1–2), 97–110.
Giuliani, M., Herman, J., Castelletti, A., and Reed, P. (2014). “Many-objective reservoir policy identification and refinement to reduce policy inertia and myopia in water management.” Water Resour. Res., 50(4), 3355–3377.
Griffin, M. T., Montz, B. E., and Arrigo, J. S. (2013). “Evaluating climate change induced water stress: A case study of the lower cape fear basin, NC.” Appl. Geogr., 40, 115–128.
Groves, D., Bloom, E., Lempert, R., Fischbach, J., Nevills, J., and Goshi, B. (2014). “Developing key indicators for adaptive water planning.” J. Water Resour. Plann. Manage., 05014008-1–05014008-10.
Groves, D. G., and Lempert, R. J. (2007). “A new analytic method for finding policy-relevant scenarios.” Global Environ. Change, 17(1), 73–85.
Haasnoot, M., Kwakkel, J. H., Walker, W. E., and ter Maat, J. (2013). “Dynamic adaptive policy pathways: A method for crafting robust decisions for a deeply uncertain world.” Global Environ. Change, 23(2), 485–498.
Hadka, D., and Reed, P. (2013). “Borg: An auto-adaptive many-objective evolutionary computing framework.” Evol. Comput., 21(2), 231–259.
Hall, J. W., Lempert, R. J., Keller, K., Hackbarth, A., Mijere, C., and McInerney, D. J. (2012). “Robust climate policies under uncertainty: A comparison of robust decision making and info-gap methods.” Risk Anal., 32(10), 1657–1672.
Hamarat, C., Kwakkel, J. H., Pruyt, E., and Loonen, E. T. (2014). “An exploratory approach for adaptive policymaking by using multi-objective robust optimization.” Simul. Modell. Pract. Theory., 46, 25–39.
Hashimoto, T., Loucks, D. P., and Stedinger, J. R. (1982). “Robustness of water resources systems.” Water Resour. Res., 18(1), 21–26.
Hawkins, E., and Sutton, R. (2009). “The potential to narrow uncertainty in regional climate predictions.” Bull. Am. Meteorol. Soc., 90(8), 1095–1107.
Hawkins, E., and Sutton, R. (2011). “The potential to narrow uncertainty in projections of regional precipitation change.” Clim. Dyn., 37(1–2), 407–418.
Herman, J. D., Kollat, J. B., Reed, P. M., and Wagener, T. (2013a). “From maps to movies: High-resolution time-varying sensitivity analysis for spatially distributed watershed models.” Hydrol. Earth Syst. Sci., 17(12), 5109–5125.
Herman, J. D., Reed, P. M., and Wagener, T. (2013b). “Time-varying sensitivity analysis clarifies the effects of watershed model formulation on model behavior.” Water Resour. Res., 49(3), 1400–1414.
Herman, J. D., Zeff, H. B., Reed, P. M., and Characklis, G. W. (2014). “Beyond optimality: Multi-stakeholder robustness tradeoffs for regional water portfolio planning under deep uncertainty.” Water Resour. Res., 50(10), 7692–7713.
Hine, D., and Hall, J. W. (2010). “Information gap analysis of flood model uncertainties and regional frequency analysis.” Water Resour. Res., 46(1).
Hipel, K. W., and Ben-Haim, Y. (1999). “Decision making in an uncertain world: Information-gap modeling in water resources management.” IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev., 29(4), 506–517.
Hitch, C. J. (1960). “On the choice of objectives in systems studies.”, Santa Monica, CA.
Hughes, J., et al. (2014). “Defining a resilient business model for water utilities.” 〈http://www.waterrf.org/PublicReportLibrary/4366.pdf〉 (Sep. 15, 2014).
Hughes, J., and Leurig, S. (2013). “Assessing water system revenue risk: Considerations for market analysts.”, Chapel Hill, NC, 〈http://www.ceres.org/resources/reports/assessing-water-system-revenue-risk-considerations-for-market-analysts〉 (Sep. 15, 2014).
Hughes, J., and Royster, S. (2014). “Overview of local government water and wastewater debt in north carolina.”, 〈http://www.efc.sog.unc.edu/sites/www.efc.sog.unc.edu/files/BorrowingForTheBigStuff_2013.pdf〉.
Hwang, C.-L., Masud, A. S. M., Paidy, S. R., and Yoon, K. P. (1979). Multiple objective decision making, methods and applications: A state-of-the-art survey, Vol. 164, Springer, Berlin.
Iooss, B., and Lemaître, P. (2014). “A review on global sensitivity analysis methods.” 〈http://arxiv.org/abs/1404.2405〉 (Apr. 24, 2014).
IPCC (Intergovernmental Panel on Climate Change). (2013). Climate Change 2013. the Physical Science Basis. Working Group I Contribution to the Fifth Assessment Rep. of the Intergovernmental Panel on Climate Change, Cambridge University Press, Cambridge U.K.
Jiménez, F., Cadenas, J. M., Sánchez, G., Gómez-Skarmeta, A. F., and Verdegay, J. L. (2006). “Multi-objective evolutionary computation and fuzzy optimization.” Int. J. Approximate Reasoning, 43(1), 59–75.
Kasprzyk, J. R., Nataraj, S., Reed, P. M., and Lempert, R. J. (2013). “Many objective robust decision making for complex environmental systems undergoing change.” Environ. Modell. Softw., 42, 55–71.
Kasprzyk, J. R., Reed, P. M., Characklis, G. W., and Kirsch, B. R. (2012). “Many-objective de novo water supply portfolio planning under deep uncertainty.” Environ. Modell. Softw., 34, 87–104.
Kasprzyk, J. R., Reed, P. M., Kirsch, B. R., and Characklis, G. W. (2009). “Managing population and drought risks using many-objective water portfolio planning under uncertainty.” Water Resour. Res., 45(12), W12401.
Knight, F. H. (1921). Risk, uncertainty and profit, Hart, Schaffner, and Marx, New York.
Korteling, B., Dessai, S., and Kapelan, Z. (2013). “Using information-gap decision theory for water resources planning under severe uncertainty.” Water Resour. Manage., 27(4), 1149–1172.
Kwakkel, J. H., and Pruyt, E. (2013). “Exploratory modeling and analysis, an approach for model-based foresight under deep uncertainty.” Technol. Forecasting Social Change, 80(3), 419–431.
Lane, M. E., Kirshen, P. H., and Vogel, R. M. (1999). “Indicators of impacts of global climate change on U.S. water resources.” J. Water Resour. Plann. Manage., 194–204.
Langlois, R. N., and Cosgel, M. M. (1993). “Frank Knight on risk, uncertainty, and the firm: A new interpretation.” Econ. Inquiry, 31(3), 456–465.
Lempert, R. J. (2002). “A new decision sciences for complex systems.” Proc. Natl. Acad. Sci., 99(suppl. 3), 7309–7313.
Lempert, R. J., Bryant, B. P., and Bankes, S. C. (2008). “Comparing algorithms for scenario discovery.”, Santa Monica, CA.
Lempert, R. J., and Collins, M. (2007). “Managing the risk of an uncertain threshold response: Comparison of robust, optimimum, and precautionary approaches.” Risk Anal., 27(4), 1009–1026.
Lempert, R. J., and Groves, D. G. (2010). “Identifying and evaluating robust adaptive policy responses to climate change for water management agencies in the American west.” Technol. Forecasting Social Change, 77(6), 960–974.
Lempert, R. J., Groves, D. G., Popper, S. W., and Bankes, S. C. (2006). “A general, analytic method for generating robust strategies and narrative scenarios.” Manage. Sci., 52(4), 514–528.
Lempert, R. J., Popper, S. W., and Bankes, S. C. (2003). Shaping the next one hundred years: New methods for quantitative, long-term policy analysis, RAND, Santa Monica, CA.
Li, K., Qi, J., Brown, C., and Ryan, J. (2014). “Effect of scenario assumptions on climate change risk estimates in a water resource system.” Clim. Res., 59(2), 149–160.
Liebman, J. C. (1976). “Some simple-minded observations on the role of optimization in public systems decision-making.” Interfaces, 6(4), 102–108.
Lins, H. F., and Stakhiv, E. Z. (1998). “Managing the nation’s water in a changing climate.” JAWRA, 34(6), 1255–1264.
Liu, Y., Gupta, H., Springer, E., and Wagener, T. (2008). “Linking science with environmental decision making: Experiences from an integrated modeling approach to supporting sustainable water resources management.” Environ. Modell. Softw., 23(7), 846–858.
Lownsbery, K. E. (2014). “Quantifying the impacts of future uncertainties on the Apalachicola-Chattahoochee-Flint basin.” M.S. thesis, University of Massachusetts, Amherst, MA.
Maass, A., Hufschmidt, M. M., Dorfman, R., Thomas, H. A., Jr., Marglin, S. A., and Fair, G. M. (1962). Design of water-resource systems: New techniques for relating economic objectives, engineering analysis, and governmental planning, Harvard University Press, Cambridge, MA.
Mahmoud, M., et al. (2009). “A formal framework for scenario development in support of environmental decision-making.” Environ. Modell. Softw., 24(7), 798–808.
Matalas, N. C., and Fiering, M. B. (1977). “Water-resource systems planning.” Climate, climatic change, and water supply, studies in geophysics, National Academy of Sciences, Washington, DC, 99–110.
Matrosov, E. S., Padula, S., and Harou, J. J. (2013a). “Selecting portfolios of water supply and demand management strategies under uncertaintycontrasting economic optimisation and ‘robust decision making’ approaches.” Water Resour. Manage., 27(4), 1123–1148.
Matrosov, E. S., Woods, A. M., and Harou, J. J. (2013b). “Robust decision making and info-gap decision theory for water resource system planning.” J. Hydrol., 494, 43–58.
McInerney, D., Lempert, R., and Keller, K. (2012). “What are robust strategies in the face of uncertain climate threshold responses?” Clim. change, 112(3–4), 547–568.
Messac, A., and Mattson, C. A. (2002). “Generating well-distributed sets of pareto points for engineering design using physical programming.” Optim. Eng., 3(4), 431–450.
Moody, P., and Brown, C. (2013). “Robustness indicators for evaluation under climate change: Application to the upper great lakes.” Water Resour. Res., 49(6), 3576–3588.
Moss, R. H., et al. (2010). “The next generation of scenarios for climate change research and assessment.” Nature, 463(7282), 747–756.
Mulvey, J. M., Vanderbei, R. J., and Zenios, S. A. (1995). “Robust optimization of large-scale systems.” Oper. Res., 43(2), 264–281.
Nazemi, A. A., and Wheater, H. S. (2014). “Assessing the vulnerability of water supply to changing streamflow conditions.” Eos, Trans. Am. Geophys. Union, 95(32), 288.
Paton, F. L., Maier, H. R., and Dandy, G. C. (2014). “Including adaptation and mitigation responses to climate change in a multiobjective evolutionary algorithm framework for urban water supply systems incorporating GHG emissions.” Water Resour. Res., 50(8), 6285–6304.
Perelman, L., Housh, M., and Ostfeld, A. (2013). “Robust optimization for water distribution systems least cost design.” Water Resour. Res., 49(10), 6795–6809.
Plischke, E., Borgonovo, E., and Smith, C. L. (2013). “Global sensitivity measures from given data.” Eur. J. Oper. Res., 226(3), 536–550.
Polasky, S., Carpenter, S. R., Folke, C., and Keeler, B. (2011). “Decision-making under great uncertainty: Environmental management in an era of global change.” Trends Ecol. Evol., 26(8), 398–404.
Postel, S. L. (2000). “Entering an era of water scarcity: The challenges ahead.” Ecol. Appl., 10(4), 941–948.
Prudhomme, C., Wilby, R. L., Crooks, S., Kay, A. L., and Reynard, N. S. (2010). “Scenario-neutral approach to climate change impact studies: Application to flood risk.” J. Hydrol., 390(3–4), 198–209.
Reed, P. M., and Kasprzyk, J. R. (2009). “Water resources management: The myth, the wicked, and the future.” J. Water Resour. Plann. Manage., 411–413.
Reuss, M. (2003). “Is it time to resurrect the Harvard water program?” J. Water Resour. Plann. Manage., 357–360.
Reusser, D., Buytaert, W., and Zehe, E. (2011). “Temporal dynamics of model parameter sensitivity for computationally expensive models with the Fourier amplitude sensitivity test.” Water Resour. Res., 47(7), W07551.
Reynoso-Meza, G., Sanchis, J., Blasco, X., and Garca-Nieto, S. (2014). “Physical programming for preference driven evolutionary multi-objective optimization.” Appl. Soft Comput., 24, 341–362.
Rittel, H. W. J., and Webber, M. M. (1973). “Dilemmas in a general theory of planning.” Pol. Sci., 4(2), 155–169.
Roy, B. (1971). “Problems and methods with multiple objective functions.” Math. Program., 1(1), 239–266.
Roy, B. (1990). “Decision-aid and decision-making.” Eur. J. Oper. Res., 45(2–3), 324–331.
Saltelli, A. (2002). “Making best use of model evaluations to compute sensitivity indices.” Comput. Phys. Commun., 145(2), 280–297.
Saltelli, A., et al. (2008). Global sensitivity analysis: The primer, Wiley, West Sussex, England.
Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M., and Tarantola, S. (2010). “Variance based sensitivity analysis of model output. design and estimator for the total sensitivity index.” Comput. Phys. Commun., 181(2), 259–270.
Saltelli, A., and Funtowicz, S. (2013). “When all models are wrong.” Issues Sci. Technol., 30(2), 79–85.
Saltelli, A., Guimarães Pereira, Â., Van der Sluijs, J. P., and Funtowicz, S. (2013). “What do I make of your latinorum? Sensitivity auditing of mathematical modelling.” Int. J. Foresight Innovation Policy, 9(2), 213–234.
Savage, L. J. (1951). “The theory of statistical decision.” J. Am. Stat. Assoc., 46(253), 55–67.
Schneller, G., and Sphicas, G. (1983). “Decision making under uncertainty: Starr’s domain criterion.” Theory Decis., 15(4), 321–336.
Schoemaker, P. J. H. (1982). “The expected utility model: Its variants, purposes, evidence and limitations.” J. Econ. Lit., 20(2), 529–563.
Scudder, T. (2005). The future of large dams, Cromwell Press, London.
Seager, R., Tzanova, A., and Nakamura, J. (2009). “Drought in the southeastern united states: Causes, variability over the last millennium, and the potential for future hydroclimate change.” J. Clim., 22(19), 5021–5045.
Simon, H. A. (1959). “Theories of decision-making in economics and behavioral science.” Am. Econ. Rev., 49(3), 253–283.
Sobol, I. (2001). “Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates.” Math. Comput. Simul., 55(1–3), 271–280.
Sobol, I. M. (1993). “Sensitivity estimates for nonlinear mathematical models and their Monte Carlo estimation.” Math. Model. Comput. Exp., 1(4), 407–417.
Starr, M. K. (1962). Product design and decision theory, Prentice-Hall, Englewood Cliffs, NJ.
Tingstad, A. H., Groves, D. G., and Lempert, R. J. (2014). “Paleoclimate scenarios to inform decision making in water resource management: Example from southern California’s inland empire.” J. Water Resour. Plann. Manage., 04014025.
Tsoukias, A. (2008). “From decision theory to decision aiding methodology.” Eur. J. Oper. Res., 187(1), 138–161.
Turner, S. W., Marlow, D., Ekström, M., Rhodes, B. G., Kularathna, U., and Jeffrey, P. J. (2014). “Linking climate projections to performance: A yield-based decision scaling assessment of a large urban water resources system.” Water Resour. Res., 50(4), 3553–3567.
Vorosmarty, C. J., Green, P., Salisbury, J., and Lammers, R. B. (2000). “Global water resources: Vulnerability from climate change and population growth.” Science, 289(5477), 284–288.
Walker, W. E., Haasnoot, M., and Kwakkel, J. H. (2013). “Adapt or perish: A review of planning approaches for adaptation under deep uncertainty.” Sustainability, 5(3), 955–979.
Walker, W. E., Rahman, S. A., and Cave, J. (2001). “Adaptive policies, policy analysis, and policy-making.” Eur. J. Oper. Res., 128(2), 282–289.
Watkins, D. W., and McKinney, D. C. (1997). “Finding robust solutions to water resources problems.” J. Water Resour. Plann. Manage., 49–58.
Weaver, C. P., Lempert, R. J., Brown, C., Hall, J. A., Revell, D., and Sarewitz, D. (2013). “Improving the contribution of climate model information to decision making: The value and demands of robust decision frameworks.” Wiley Interdiscip. Rev. Clim. Change, 4(1), 39–60.
Woodruff, M. J., Reed, P. M., and Simpson, T. (2013). “Many objective visual analytics: Rethinking the design of complex engineered systems.” Struct. Multidiscip. Optim., 48(1), 201–219.
Zeff, H. B., and Characklis, G. W. (2013). “Managing water utility financial risks through third-party index insurance contracts.” Water Resour. Res., 49(8), 4939–4951.
Zeff, H. B., Kasprzyk, J. R., Herman, J. D., Reed, P. M., and Characklis, G. W. (2014). “Navigating financial and supply reliability tradeoffs in regional drought portfolios.” Water Resour. Res., 50(6), 4906–4923.
Information & Authors
Information
Published In
Journal of Water Resources Planning and Management
Volume 141 • Issue 10 • October 2015
Copyright
© 2015 American Society of Civil Engineers.
History
Received: Sep 19, 2014
Accepted: Dec 3, 2014
Published online: Feb 10, 2015
Discussion open until: Jul 10, 2015
Published in print: Oct 1, 2015
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.