Curses, Tradeoffs, and Scalable Management: Advancing Evolutionary Multiobjective Direct Policy Search to Improve Water Reservoir Operations
Publication: Journal of Water Resources Planning and Management
Volume 142, Issue 2
Abstract
Optimal management policies for water reservoir operation are generally designed via stochastic dynamic programming (SDP). Yet, the adoption of SDP in complex real-world problems is challenged by the three curses of dimensionality, modeling, and multiple objectives. These three curses considerably limit SDP’s practical application. Alternatively, this study focuses on the use of evolutionary multiobjective direct policy search (EMODPS), a simulation-based optimization approach that combines direct policy search, nonlinear approximating networks, and multiobjective evolutionary algorithms to design Pareto-approximate closed-loop operating policies for multipurpose water reservoirs. This analysis explores the technical and practical implications of using EMODPS through a careful diagnostic assessment of the effectiveness and reliability of the overall EMODPS solution design as well as of the resulting Pareto-approximate operating policies. The EMODPS approach is evaluated using the multipurpose Hoa Binh water reservoir in Vietnam, where water operators are seeking to balance the conflicting objectives of maximizing hydropower production and minimizing flood risks. A key choice in the EMODPS approach is the selection of alternative formulations for flexibly representing reservoir operating policies. This study distinguishes between the relative performance of two widely-used nonlinear approximating networks, namely artificial neural networks (ANNs) and radial basis functions (RBFs). The results show that RBF solutions are more effective than ANN ones in designing Pareto approximate policies for the Hoa Binh reservoir. Given the approximate nature of EMODPS, the diagnostic benchmarking uses SDP to evaluate the overall quality of the attained Pareto-approximate results. Although the Hoa Binh test case’s relative simplicity should maximize the potential value of SDP, the results demonstrate that EMODPS successfully dominates the solutions derived via SDP.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
This work was partially supported by the IMRR-Integrated and sustainable water management of the Red-Thai Binh Rivers System in changing climate research project funded by the Italian Ministry of Foreign Affair as part of its development cooperation program. Francesca Pianosi was supported by the Natural Environment Research Council (Consortium on Risk in the Environment: Diagnostics, Integration, Benchmarking, Learning and Elicitation (CREDIBLE); grant number NE/J017450/1).
References
Ansar, A., Flyvbjerg, B., Budzier, A., and Lunn, D. (2014). “Should we build more large dams? The actual costs of hydropower megaproject development.” Energy Policy, 69, 43–56.
Baxter, J., Bartlett, P., and Weaver, L. (2001). “Experiments with infinite-horizon, policy-gradient estimation.” J. Artif. Intell. Res., 15, 351–381.
Bellman, R. (1957). Dynamic programming, Princeton University Press, Princeton, NJ.
Bertsekas, D. (1976). Dynamic programming and stochastic control, Academic Press, New York.
Bertsekas, D. (2005). “Dynamic programming and suboptimal control: A survey from ADP to MPC.” Eur. J. Control, 11(4–5), 310–334.
Bertsekas, D., and Tsitsiklis, J. (1996). Neuro-dynamic programming, Athena Scientific, Belmont, MA.
Biglarbeigi, P., Giuliani, M., and Castelletti, A. (2014). “Many-objective direct policy search in the Dez and Karoun multireservoir system, Iran.” Proc., World Environmental and Water Resources Congress (ASCE EWRI 2014), Portland, OR.
Brill, E., Flach, J., Hopkins, L., and Ranjithan, S. (1990). “MGA: A decision support system for complex, incompletely defined problems.” IEEE Trans. Syst. Man Cybern., 20(4), 745–757.
Buhmann, M. (2003). Radial basis functions: Theory and implementations, Cambridge University Press, Cambridge, U.K.
Busoniu, L., Babuska, R., De Schutter, B., and Ernst, D. (2010). Reinforcement learning and dynamic programming using function approximators, CRC Press, New York.
Busoniu, L., Ernst, D., De Schutter, B., and Babuska, R. (2011). “Cross-entropy optimization of control policies with adaptive basis functions.” IEEE Trans. Syst. Man Cybern. Part B Cybern., 41(1), 196–209.
Castelletti, A., de Rigo, D., Rizzoli, A., Soncini-Sessa, R., and Weber, E. (2007). “Neuro-dynamic programming for designing water reservoir network management policies.” Control Eng. Pract., 15(8), 1031–1038.
Castelletti, A., Galelli, S., Restelli, M., and Soncini-Sessa, R. (2010). “Tree-based reinforcement learning for optimal water reservoir operation.” Water Resour. Res., 46(9), W09507.
Castelletti, A., Pianosi, F., Quach, X., and Soncini-Sessa, R. (2012). “Assessing water reservoirs management and development in northern Vietnam.” Hydrol. Earth Syst. Sci., 16(1), 189–199.
Castelletti, A., Pianosi, F., and Restelli, M. (2013). “A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run.” Water Resour. Res., 49, 3476–3486.
Castelletti, A., Pianosi, F., and Soncini-Sessa, R. (2008). “Water reservoir control under economic, social and environmental constraints.” Automatica, 44(6), 1595–1607.
Castelletti, A., Pianosi, F., and Soncini-Sessa, R. (2012a). “Stochastic and robust control of water resource systems: Concepts, methods and applications.” System identification, environmental modelling, and control system design, Springer, London, 383–401.
Castelletti, A., Pianosi, F., Quach, X., and Soncini-Sessa, R. (2012b). “Assessing water reservoirs management and development in northern Vietnam.” Hydrol. Earth Syst. Sci., 16(1), 189–199.
Celeste, A., and Billib, M. (2009). “Evaluation of stochastic reservoir operation optimization models.” Adv. Water Resour., 32(9), 1429–1443.
Chankong, V., and Haimes, Y. (1983). Multiobjective decision making: Theory and methodology, North-Holland, New York.
Chen, T., and Chen, H. (1995). “Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems.” IEEE Trans. Neural Networks, 6(4), 911–917.
Clark, E. (1950). “New York control curves.” J. Am. Water Works Assoc., 42(9), 823–827.
Clark, E. (1956). “Impounding reservoirs.” J. Am. Water Works Assoc., 48(4), 349–354.
Coello Coello, C., Lamont, G., and Veldhuizen, D. V. (2007). Evolutionary algorithms for solving multiobjective problems (genetic algorithms and evolutionary computation), 2nd Ed., Springer, New York.
Cohon, J. L., and Marks, D. (1975). “A review and evaluation of multiobjective programing techniques.” Water Resour. Res., 11(2), 208–220.
Cui, L., and Kuczera, G. (2005). “Optimizing water supply headworks operating rules under stochastic inputs: Assessment of genetic algorithm performance.” Water Resour. Res., 41, W05016.
Cybenko, G. (1989). “Approximation by superpositions of a sigmoidal function.” Math. Control Signals Syst., 2(4), 303–314.
Dariane, A., and Momtahen, S. (2009). “Optimization of multireservoir systems operation using modified direct search genetic algorithm.” J. Water Resour. Plann. Manage., 141–148.
Deb, K. (2001). Multiobjective optimization using evolutionary algorithms, Wiley, Hoboken, NJ.
Deisenroth, M., Neumann, G., and Peters, J. (2011). “A survey on policy search for robotics.” Found. Trends Rob., 2(1–2), 1–142.
de Rigo, D., Castelletti, A., Rizzoli, A., Soncini-Sessa, R., and Weber, E. (2005). “A selective improvement technique for fastening neuro-dynamic programming in water resources network management.” Proc., 16th IFAC World Congress, Prague, Czech Republic.
Desreumaux, Q., Côté, P., and Leconte, R. (2014). “Role of hydrologic information in stochastic dynamic programming: A case study of the Kemano hydropower system in British Columbia.” Can. J. Civ. Eng., 41(9), 839–844.
Draper, A., and Lund, J. (2004). “Optimal hedging and carryover storage value.” J. Water Resour. Plann. Manage., 83–87.
Esogbue, A. (1989). “Dynamic programming and water resources: Origins and interconnections.” Dynamic programming for optimal water resources systems analysis, Prentice-Hall, Englewood Cliffs, NJ.
Faber, B., and Stedinger, J. (2001). “Reservoir optimization using sampling SDP with ensemble streamflow prediction (ESP) forecasts.” J. Hydrol., 249(1), 113–133.
Fleming, P., Purshouse, R., and Lygoe, R. (2005). “Many-objective optimization: An engineering design perspective.” Proc., 3rd Int. Conf. on Evolutionary Multi-Criterion Optimization, Guanajuato, Mexico, 14–32.
Funahashi, K. (1989). “On the approximate realization of continuous mappings by neural networks.” Neural Networks, 2(3), 183–192.
Gaggero, M., Gnecco, G., and Sanguineti, M. (2014). “Suboptimal policies for stochastic N-stage optimization: Accuracy analysis and a case study from optimal consumption.” Models and methods in economics and management science, F. E. Ouardighi and K. Kogan, eds., Springer, New York, 27–50.
Gass, S., and Saaty, T. (1955). “Parametric objective function—Part II.” Oper. Res., 3, 316–319.
Giuliani, M., Galelli, S., and Soncini-Sessa, R. (2014a). “A dimensionality reduction approach for many-objective Markov decision processes: Application to a water reservoir operation problem.” Environ. Modell. Software, 57, 101–114.
Giuliani, M., Herman, J., Castelletti, A., and Reed, P. (2014b). “Many-objective reservoir policy identification and refinement to reduce policy inertia and myopia in water management.” Water Resour. Res., 50(4), 3355–3377.
Giuliani, M., Mason, E., Castelletti, A., Pianosi, F., and Soncini-Sessa, R. (2014c). “Universal approximators for direct policy search in multi-purpose water reservoir management: A comparative analysis.” Proc., 19th IFAC World Congress, Cape Town, South Africa.
Gleick, P., and Palaniappan, M. (2010). “Peak water limits to freshwater withdrawal and use.” Proc. Natl. Acad. Sci. U.S.A., 107(25), 11155–11162.
Guariso, G., Rinaldi, S., and Soncini-Sessa, R. (1986). “The management of Lake Como: A multiobjective analysis.” Water Resour. Res., 22(2), 109–120.
Guo, X., Hu, T., Zeng, X., and Li, X. (2013). “Extension of parametric rule with the hedging rule for managing multireservoir system during droughts.” J. Water Resour. Plann. Manage., 139–148.
Hadka, D., and Reed, P. (2012). “Diagnostic assessment of search controls and failure modes in many-objective evolutionary optimization.” Evol. Comput., 20(3), 423–452.
Hadka, D., and Reed, P. (2013). “Borg: An auto-adaptive many-objective evolutionary computing framework.” Evol. Comput., 21(2), 231–259.
Haimes, Y., Lasdon, L., and Wismer, D. (1971). “On a bicriterion formulation of the problems of integrated system identification and system optimization.” IEEE Trans. Syst. Man Cybern., 1(3), 296–297.
Hall, W., and Buras, N. (1961). “The dynamic programming approach to water-resources development.” J. Geophys. Res., 66(2), 517–520.
Heidrich-Meisner, V., and Igel, C. (2008). “Variable metric reinforcement learning methods applied to the noisy mountain car problem.” Recent advances in reinforcement learning, Springer, Berlin, 136–150.
Hejazi, M., and Cai, X. (2009). “Input variable selection for water resources systems using a modified minimum redundancy maximum relevance (mMRMR) algorithm.” Adv. Water Resour., 32(4), 582–593.
Herman, J., Reed, P., Zeff, H., and Characklis, G. (2015). “How should robustness be defined for water systems planning under change?” J. Water Resour. Plann. Manage., 04015012.
Hornik, K., Stinchcombe, M., and White, H. (1989). “Multilayer feedforward networks are universal approximators.” Neural Networks, 2(5), 359–366.
Kasprzyk, J., Reed, P., Kirsch, B., and Characklis, G. (2009). “Managing population and drought risks using many-objective water portfolio planning under uncertainty.” Water Resour. Res., 45(12).
Knowles, J., and Corne, D. (2002). “On metrics for comparing non–dominated sets.” Proc., 2002 World Congress on Computational Intelligence (WCCI), IEEE Computer Society, 711–716.
Koutsoyiannis, D., and Economou, A. (2003). “Evaluation of the parameterization-simulation-optimization approach for the control of reservoir systems.” Water Resour. Res., 39(6).
Labadie, J. (2004). “Optimal operation of multireservoir systems: State-of-the-art review.” J. Water Resour. Plann. Manage., 93–111.
Loucks, D., and Sigvaldason, O. (1982). “Multiple-reservoir operation in North America.” The operation of multiple reservoir systems, Z. Kaczmarck and J. Kindler, eds., IIASA Collaborative Proceedings Series, 1–103.
Loucks, D., van Beek, E., Stedinger, J., Dijkman, J., and Villars, M. (2005). Water resources systems planning and management: An introduction to methods, models and applications, UNESCO, Paris.
Lund, J., and Guzman, J. (1999). “Derived operating rules for reservoirs in series or in parallel.” J. Water Resour. Plann. Manage., 143–153.
Maass, A., Hufschmidt, M., Dorfman, R., Thomas, H., Jr., Marglin, S., and Fair, G. (1962). Design of water–resource systems, Harvard University Press, Cambridge, MA.
Maier, H., et al. (2014). “Evolutionary algorithms and other metaheuristics in water resources: Current status, research challenges and future directions.” Environ. Modell. Software, 62, 271–299.
Maier, H., and Dandy, G. (2000). “Neural networks for the prediction and forecasting of water resources variables: A review of modelling issues and applications.” Environ. Modell. Software, 15(1), 101–124.
Marbach, P., and Tsitsiklis, J. (2001). “Simulation-based optimization of Markov reward processes.” IEEE Trans. Autom. Control, 46(2), 191–209.
Mayne, D. Q., Rawlings, J. B., Rao, C. V., and Scokaert, P. O. (2000). “Constrained model predictive control: Stability and optimality.” Automatica, 36(6), 789–814.
McDonald, R. I., et al. (2011). “Urban growth, climate change, and freshwater availability.” Proc. Natl. Acad. Sci., 108(15), 6312–6317.
Mhaskar, H., and Micchelli, C. (1992). “Approximation by superposition of sigmoidal and radial basis functions.” Adv. Appl. Math., 13(3), 350–373.
Momtahen, S., and Dariane, A. (2007). “Direct search approaches using genetic algorithms for optimization of water reservoir operating policies.” J. Water Resour. Plann. Manage., 202–209.
Moriarty, D., Schultz, A., and Grefenstette, J. (1999). “Evolutionary algorithms for reinforcement learning.” J. Artif. Intell. Res., 11, 199–229.
Nalbantis, I., and Koutsoyiannis, D. (1997). “A parametric rule for planning and management of multiple-reservoir systems.” Water Resour. Res., 33(9), 2165–2177.
Nicklow, J., et al. (2010). “State of the art for genetic algorithms and beyond in water resources planning and management.” J. Water Resour. Plann. Manage., 412–432.
Oliveira, R., and Loucks, D. P. (1997). “Operating rules for multi reservoir systems.” Water Resour. Res., 33(4), 839–852.
Park, J., and Sandberg, I. (1991). “Universal approximation using radial-basis-function networks.” Neural Comput., 3(2), 246–257.
Peters, J., and Schaal, S. (2008). “Reinforcement learning of motor skills with policy gradients.” Neural Networks, 21(4), 682–697.
Pianosi, F., Quach, X., and Soncini-Sessa, R. (2011). “Artificial neural networks and multi objective genetic algorithms for water resources management: An application to the Hoa Binh reservoir in Vietnam.” Proc., 18th IFAC World Congress, Milan, Italy.
Piccardi, C., and Soncini-Sessa, R. (1991). “Stochastic dynamic programming for reservoir optimal control: Dense discretization and inflow correlation assumption made possible by parallel computing.” Water Resour. Res., 27(5), 729–741.
Powell, W. (2007). Approximate dynamic programming: Solving the curses of dimensionality, Wiley, Hoboken, NJ.
Reed, P., Hadka, D., Herman, J., Kasprzyk, J., and Kollat, J. (2013). “Evolutionary multiobjective optimization in water resources: The past, present, and future.” Adv. Water Resour., 51, 438–456.
Reed, P. M., and Kollat, J. B. (2013). “Visual analytics clarify the scalability and effectiveness of massively parallel many-objective optimization: A groundwater monitoring design example.” Adv. Water Resour., 56, 1–13.
ReVelle, C., and McGarity, A. E. (1997). Design and operation of civil and environmental engineering systems, Wiley, Hoboken, NJ.
Rippl, W. (1883). “The capacity of storage reservoirs for water supply.” Minutes Proc., 71, 270–278.
Rosenstein, M., and Barto, A. (2001). “Robot weightlifting by direct policy search.” Int. Joint Conf. on Artificial Intelligence, Citeseer, 839–846.
Sehnke, F., Osendorfer, C., Rückstieß, T., Graves, A., Peters, J., and Schmidhuber, J. (2010). “Parameter-exploring policy gradients.” Neural Networks, 23(4), 551–559.
Soncini-Sessa, R., Castelletti, A., and Weber, E. (2007). Integrated and participatory water resources management: Theory, Elsevier, Amsterdam, Netherlands.
Sutton, R., McAllester, D., Singh, S., and Mansour, Y. (2000). “Policy gradient methods for reinforcement learning with function approximation.” Adv. Neural Inf. Process. Syst., 12, 1057–1063.
Tejada-Guibert, J., Johnson, S., and Stedinger, J. (1995). “The value of hydrologic information in stochastic dynamic programming models of a multireservoir system.” Water Resour. Res., 31(10), 2571–2579.
Tikk, D., Kóczy, L., and Gedeon, T. (2003). “A survey on universal approximation and its limits in soft computing techniques.” Int. J. Approximate Reasoning, 33(2), 185–202.
Tsitsiklis, J., and Van Roy, B. (1996). “Feature-based methods for large scale dynamic programming.” Mach. Learn., 22, 59–94.
Tu, M., Hsu, N., and Yeh, W. (2003). “Optimization of reservoir management and operation with hedging rules.” J. Water Resour. Plann. Manage., 86–97.
U.S. Army Corps of Engineers. (1977). “Reservoir system analysis for conservation, hydrologic engineering methods for water resources development.” Hydrologic Engineering Center, Davis, CA.
Whiteson, S., and Stone, P. (2006). “Evolutionary function approximation for reinforcement learning.” J. Mach. Learn. Res., 7, 877–917.
Whitley, D., Dominic, S., Das, R., and Anderson, C. (1994). Genetic reinforcement learning for neurocontrol problems, Springer, New York.
Woodruff, M., Reed, P., and Simpson, T. (2013). “Many objective visual analytics: Rethinking the design of complex engineered systems.” Struct. Multidiscip. Optim., 48(1), 201–219.
Yeh, W. (1985). “Reservoir management and operations models: A state of the art review.” Water Resour. Res., 21(12), 1797–1818.
Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C., and da Fonseca, V. (2003). “Performance assessment of multiobjective optimizers: An analysis and review.” IEEE Trans. Evol. Comput., 7(2), 117–132.
Zoppoli, R., Sanguineti, M., and Parisini, T. (2002). “Approximating networks and extended ritz method for the solution of functional optimization problems.” J. Optim. Theory Appl., 112(2), 403–440.
Information & Authors
Information
Published In
Journal of Water Resources Planning and Management
Volume 142 • Issue 2 • February 2016
Copyright
© 2015 American Society of Civil Engineers.
History
Received: Nov 11, 2014
Accepted: May 28, 2015
Published online: Aug 13, 2015
Discussion open until: Jan 13, 2016
Published in print: Feb 1, 2016
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.