Deep Reinforcement Learning for Real-Time Optimization of Pumps in Water Distribution Systems
Publication: Journal of Water Resources Planning and Management
Volume 146, Issue 11
Abstract
Real-time control of pumps can be an infeasible task in water distribution systems (WDSs) because the calculation to find the optimal pump speeds is resource intensive. The computational need cannot be lowered even with the capabilities of smart water networks when conventional optimization techniques are used. Deep reinforcement learning (DRL) is presented here as a controller of pumps in two WDSs. An agent based on a dueling deep q-network is trained to maintain the pump speeds based on instantaneous nodal pressure data. General optimization techniques (e.g., Nelder–Mead method, differential evolution) serve as baselines. The total efficiency achieved by the DRL agent compared to the best-performing baseline is above 0.98, whereas the speedup is around compared to that. The main contribution of the presented approach is that the agent can run the pumps in real time because it depends only on measurement data. If the WDS is replaced with a hydraulic simulation, the agent still outperforms conventional techniques in search speed.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code generated or used during the study are available in a repository online in accordance with funder data retention policies [Reinforcement learning algorithms implementation (Hill et al. 2018), Python API for EPANET (Heinsbroek 2016), original numerical models of Anytown and D-Town (University of Exeter 1986), the code repository of the presented research, including everything to reproduce the results (Hajgató et al. 2020).]
Acknowledgments
The research presented in this paper has been supported by the BME-Artificial Intelligence FIKP Grant of Ministry of Human Resources (BME FIKP-MI/SC) and by the National Research, Development and Innovation Fund (TUDFO/51757/2019-ITM), Thematic Excellence Program. Bálint Gyires-Tóth is grateful for the financial support of the Doctoral Research Scholarship of Ministry of Human Resources (ÚNKP-19-4-BME-189) in the scope of New National Excellence Program and of János Bolyai Research Scholarship of the Hungarian Academy of Sciences.
References
Abdallah, M., and Z. Kapelan. 2017. “Iterative extended lexicographic goal programming method for fast and optimal pump scheduling in water distribution networks.” J. Water Resour. Plann. Manage. 143 (11): 04017066. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000843.
Abdallah, M., and Z. Kapelan. 2019. “Fast pump scheduling method for optimum energy cost and water quality in water distribution networks with fixed and variable speed pumps.” J. Water Resour. Plann. Manage. 145 (12): 04019055. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001123.
Akiba, T., S. Sano, T. Yanase, T. Ohta, and M. Koyama. 2019. “Optuna.” In Proc., 25th ACM SIGKDD Int. Conf. on Knowledge Discovery & Data Mining—KDD ’19. New York: ACM Press.
Baltar, A. M., and D. G. Fontane. 2008. “Use of multiobjective particle swarm optimization in water resources management.” J. Water Resour. Plann. Manage. 134 (3): 257–265. https://doi.org/10.1061/(ASCE)0733-9496(2008)134:3(257).
Bene, J. G. 2013. “Pump schedule optimisation techniques for water distribution systems.” Ph.D. thesis, Dept. of Process and Environmental Engineering, Univ. of Oulu Graduate School.
Bergstra, J. S., R. Bardenet, Y. Bengio, and B. Kégl. 2011. “Algorithms for hyper-parameter optimization.” In Proc., Advances in Neural Information Processing Systems 24, edited by J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, 2546–2554. New York: Curran Associates.
Bhattacharya, B., A. H. Lobbrecht, and D. P. Solomatine. 2003. “Neural networks and reinforcement learning in control of water systems.” J. Water Resour. Plann. Manage. 129 (6): 458–465. https://doi.org/10.1061/(ASCE)0733-9496(2003)129:6(458).
Bottou, L., and O. Bousquet. 2008. “The tradeoffs of large scale learning.” In Proc., Advances in Neural Information Processing Systems 20 (NIPS 2007), edited by J. Platt, D. Koller, Y. Singer, and S. Roweis, 161–168. San Diego: Neural Information Processing Systems Foundation.
Castelletti, A., F. Pianosi, and M. Restelli. 2013. “A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run.” Water Resour. Res. 49 (6): 3476–3486. https://doi.org/10.1002/wrcr.20295.
Celi, C. F. L., P. L. Iglesias-Rey, and F. J. M. Solano. 2017. “Energy optimization of supplied flows from multiple pumping stations in water distributions networks.” Procedia Eng. 186 (1): 93–100. https://doi.org/10.1016/j.proeng.2017.03.214.
Cybenko, G. 1989. “Approximation by superpositions of a sigmoidal function.” Math. Control Signals Syst. 2 (4): 303–314. https://doi.org/10.1007/BF02551274.
Guo, G., S. Liu, Y. Wu, J. Li, R. Zhou, and X. Zhu. 2018. “Short-term water demand forecast based on deep learning method.” J. Water Resour. Plann. Manage. 144 (12): 04018076. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000992.
Hajgató, G., B. Gyires-Tóth, and G. Paál. 2020. “RL-WDS.” Accessed February 14, 2020. https://github.com/BME-SmartLab/rl-wds.
Heinsbroek, A. 2016. “EPYNET.” Accessed February 14, 2020. https://github.com/Vitens/epynet.
Hill, A., et al. 2018. “Stable baselines.” Accessed February 14, 2020. https://github.com/hill-a/stable-baselines.
Lehman, J., et al. 2018. “The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities.” Preprint, submitted March 9, 2018. https://arxiv.org/abs/1803.03453v3.
Machell, J., S. R. Mounce, and J. B. Boxall. 2010. “Online modelling of water distribution systems: A UK case study.” Drinking Water Eng. Sci. 3 (1): 21–27. https://doi.org/10.5194/dwes-3-21-2010.
Mahootchi, M., H. R. Tizhoosh, and K. Ponnambalam. 2007. “Reservoir operation optimization by reinforcement learning.” J. Water Manage. Model. 15 (1): 165–184. https://doi.org/10.14796/JWMM.R227-08.
Mala-Jetmarova, H., N. Sultanova, and D. Savic. 2018. “Lost in optimisation of water distribution systems? A literature review of system design.” Water 10 (3): 307. https://doi.org/10.3390/w10030307.
Manheim, D., and S. Garrabrant. 2018. “Categorizing variants of Goodhart’s law.” Preprint, submitted March 13, 2018. https://arxiv.org/abs/1803.04585.
McClymont, K., E. Keedwell, D. Savić, and M. Randall-Smith. 2013. “A general multi-objective hyper-heuristic for water distribution network design with discolouration risk.” J. Hydroinf. 15 (3): 700–716. https://doi.org/10.2166/hydro.2012.022.
Mnih, V., K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. 2013. “Playing Atari with deep reinforcement learning.” Preprint, submitted December 19, 2013. https://arxiv.org/abs/1312.5602.
Nair, V., and G. E. Hinton. 2010. “Rectified linear units improve restricted Boltzmann machines.” In Proc., 27th Int. Conf. on Machine Learning, ICML’10, 807–814. Brookline, MA: Microtome Publishing. http://dl.acm.org/citation.cfm?id=3104322.3104425.
Nardo, A. D., et al. 2018. “On-line measuring sensors for smart water network monitoring.” In Proc., 13th Int. Conf. on Hydroinformatics. Palermo, Italy: Univ. of Palermo.
Odan, F. K., L. F. R. Reis, and Z. Kapelan. 2015. “Real-time multiobjective optimization of operation of water supply systems.” J. Water Resour. Plann. Manage. 141 (9): 04015011. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000515.
Pascual, J., J. Romera, V. Puig, R. Creus, and M. Minoves. 2011. “Operational predictive optimal control of Barcelona water transport network.” IFAC Proc. Volumes 44 (1): 10571–10578. https://doi.org/10.3182/20110828-6-IT-1002.01079.
Razavi, S., B. A. Tolson, and D. H. Burn. 2012. “Review of surrogate modeling in water resources.” Water Resour. Res. 48 (7): W07401. https://doi.org/10.1029/2011WR011527.
Ritter, J., G. Corzo, D. P. Solomatine, and H. Angarita. 2020. “Multiobjective direct policy search using physically based operating rules in multireservoir systems.” J. Water Resour. Plann. Manage. 146 (4): 05020002. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001159.
Rossmann, L. A. 1993. “EPANET: An advanced water quality modeling package for distribution systems.” In Proc., Annual Conf.—American Water Works Association, 411–418. Washington, DC: USEPA.
Savic, D., H. Mala-Jetmarova, and N. Sultanova. 2018. “History of optimization in water distribution system analysis.” In Vol. 1 of Proc., WDSA/CCWI Joint Conf. Boca Raton, FL: CRC Press.
Simpson, T. W., A. J. Booker, D. Ghosh, A. A. Giunta, P. N. Koch, and R.-J. Yang. 2004. “Approximation methods in multidisciplinary analysis and optimization: A panel discussion.” Struct. Multidiscip. Optim. 27 (5): 302–313. https://doi.org/10.1007/s00158-004-0389-9.
Sutton, R. S., and A. G. Barto. 2018. Reinforcement learning. Cambridge, MA: MIT Press.
Taormina, R., and S. Galelli. 2018. “Deep-learning approach to the detection and localization of cyber-physical attacks on water distribution systems.” J. Water Resour. Plann. Manage. 144 (10): 04018065. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000983.
United Nations. 2018. “World urbanization prospects: The 2018 revision.” Accessed February 14, 2020. https://population.un.org/wup/Publications/Files/WUP2018-Report.pdf.
United Nations. 2019. “World population prospects 2019: Data booklet.” Accessed February 14, 2020. https://population.un.org/wpp/Publications/Files/WPP2019_DataBooklet.pdf.
University of Exeter. 1986. “Centre for water systems.” Accessed February 14, 2020. http://emps.exeter.ac.uk/engineering/research/cws/.
Wang, Y., G. Cembrano, V. Puig, M. Urrea, J. Romera, D. Saporta, J. G. Valero, and J. Quevedo. 2017. “Optimal management of Barcelona water distribution network using non-linear model predictive control.” IFAC-PapersOnLine 50 (1): 5380–5385. https://doi.org/10.1016/j.ifacol.2017.08.1070.
Wang, Z., T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas. 2015. “Dueling network architectures for deep reinforcement learning.” Preprint, submitted November 20, 2015. https://arxiv.org/abs/1511.06581.
Zessler, U., and U. Shamir. 1989. “Optimal operation of water distribution systems.” J. Water Resour. Plann. Manage. 115 (6): 735–752. https://doi.org/10.1061/(ASCE)0733-9496(1989)115:6(735).
Information & Authors
Information
Published In
Journal of Water Resources Planning and Management
Volume 146 • Issue 11 • November 2020
Copyright
© 2020 American Society of Civil Engineers.
History
Received: Oct 17, 2019
Accepted: May 26, 2020
Published online: Aug 25, 2020
Published in print: Nov 1, 2020
Discussion open until: Jan 25, 2021
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.