Optimal Control of Combined Sewer Systems to Minimize Sewer Overflows by Using Reinforcement Learning
Publication: World Environmental and Water Resources Congress 2023
ABSTRACT
A combined sewer system (CSS) collects rainwater runoff, domestic sewage, and industrial wastewater in the same pipe. The volume of wastewater can sometimes exceed the system capacity during heavy rainfall events. When this occurs, untreated stormwater and wastewater discharge directly to nearby streams, rivers, and other water bodies. This would threaten public health and the environment, contributing to drinking water contamination and other concerns. Minimizing sewer overflows requires an optimization method that can provide an optimal sequence of decision variables at control gates. Conventional strategies use classical optimization algorithms, such as genetic algorithms and pattern search, to find the optimal sequence of decision variables. However, these conventional frameworks are very time-consuming, and it is almost impossible to achieve near real-time optimal control. This paper presents a faster optimization framework by using a new optimal control tool: reinforcement learning. The environment (flow modeler) used in this paper is the numerical model: Environmental Protection Agency’s Storm Water Management Model (EPA SWMM) to ensure the accuracy of environment response. The reward function is constructed based on the calculated water depth and overflow rate from SWMM. The process keeps minimizing the reward function to obtain the optimal flow release sequence at each controlled orifice gate. The combined sewer system (CSS) of the Puritan-Fenkell 7-mile facility in Detroit, MI, is chosen as the case study.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Abdalla, E. M. H., Pons, V., Stovin, V., De-Ville, S., Fassman-Beck, E., Alfredsen, K., and Muthanna, T. M. (2021). Evaluating different machine learning methods to simulate runoff from extensive green roofs. Hydrology and Earth System Sciences Discussions, 1–24.
Adnan, R. M., Petroselli, A., Heddam, S., Santos, C. A. G., and Kisi, O. (2021). Short term rainfall-runoff modelling using several machine learning methods and a conceptual event-based model. Stochastic Environmental Research and Risk Assessment, 35(3), 597–616.
Bowes, B. D., Tavakoli, A., Wang, C., Heydarian, A., Behl, M., Beling, P. A., and Goodall, J. L. (2021). Flood mitigation in coastal urban catchments using real-time stormwater infrastructure control and reinforcement learning. Journal of Hydroinformatics, 23(3), 529–547.
Chen, J. X. (2016). The evolution of computing: AlphaGo. Computing in Science & Engineering, 18(4), 4–7.
Delipetrev, B., Jonoski, A., and Solomatine, D. P. (2017). A novel nested stochastic dynamic programming (nSDP) and nested reinforcement learning (nRL) algorithm for multipurpose reservoir optimization. Journal of Hydroinformatics, 19(1), 47–61.
Ezer, T., and Atkinson, L. P. (2014). Accelerated flooding along the US East Coast: On the impact of sea‐level rise, tides, storms, the Gulf Stream, and the North Atlantic Oscillations. Earth’s Future, 2(8), 362–382.
Hajgató, G., Paál, G., and Gyires-Tóth, B. (2020). Deep reinforcement learning for real-time optimization of pumps in water distribution systems. arXiv preprint arXiv:2010.06460.
Jin, A., Fox-Lent, C., and Linkov, I. (2020). Resilience for smart water systems. J. Water Resour. Plann. Manage, 146(1), 02519002.
Kerkez, B., Gruden, C., Lewis, M., Montestruque, L., Quigley, M., Wong, B., and Pak, C. (2016). Smarter stormwater systems.
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A. A., Yogamani, S., and Pérez, P. (2021). Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems.
Leon, A. S., Tang, Y., Qin, L., and Chen, D. (2020). A MATLAB framework for forecasting optimal flow releases in a multi-storage system for flood control. Environmental Modelling & Software, 125, 104618.
Leon, A. S., Bian, L., and Tang, Y. (2021). Comparison of the genetic algorithm and pattern search methods for forecasting optimal flow releases in a multi-storage system for flood control. Environmental Modelling & Software, 145, 105198.
Li, P., Zhang, J., and Krebs, P. (2022). Prediction of Flow Based on a CNN-LSTM Combined Deep Learning Approach. Water, 14(6), 993.
Miller, J. D., and Hutchins, M. (2017). The impacts of urbanisation and climate change on urban flooding and urban water quality: A review of the evidence concerning the United Kingdom. Journal of Hydrology: Regional Studies, 12, 345–362.
Mounce, S. R., Shepherd, W., Ostojin, S., Abdel-Aal, M., Schellart, A. N. A., Shucksmith, J. D., and Tait, S. J. (2020). Optimisation of a fuzzy logic-based local real-time control system for mitigation of sewer flooding using genetic algorithms. Journal of Hydroinformatics, 22(2), 281–295.
Mullapudi, A., Wong, B. P., and Kerkez, B. (2017). Emerging investigators series: building a theory for smart stormwater systems. Environmental Science: Water Research & Technology, 3(1), 66–77.
Mullapudi, A., and Kerkez, B. (2018). Autonomous control of urban storm water networks using reinforcement learning. EPiC Series in Engineering, 3, 1465–1469.
Mynett, A. E., and Vojinovic, Z. (2009). Hydroinformatics in multi-colours—Part red: Urban flood and disaster management. Journal of Hydroinformatics, 11(3-4), 166–180.
NASA. (2017). Earth observatory: how will global warming change earth? URL: https://earthobservatory.nasa.gov/Features/GlobalWarming/page6.php (2017).
Pianosi, F., Castelletti, A., and Restelli, M. (2013). Tree-based fitted Q-iteration for multi-objective Markov decision processes in water resource management. Journal of Hydroinformatics, 15(2), 258–270.
Sadler, J. M., Goodall, J. L., Morsy, M. M., and Spencer, K. (2018). Modeling urban coastal flood severity from crowd-sourced flood reports using Poisson regression and Random Forest. Journal of hydrology, 559, 43–55.
Sajedi‐Hosseini, F., Choubin, B., Solaimani, K., Cerdà, A., and Kavian, A. (2018). Spatial prediction of soil erosion susceptibility using a fuzzy analytical network process: Application of the fuzzy decision making trial and evaluation laboratory approach. Land degradation & development, 29(9), 3092–3103.
Saliba, S. M., Bowes, B. D., Adams, S., Beling, P. A., and Goodall, J. L. (2020). Deep reinforcement learning with uncertain data for real-time stormwater system control and flood mitigation. Water, 12(11), 3222.
Sallab, A. E., Abdou, M., Perot, E., and Yogamani, S. (2017). Deep reinforcement learning framework for autonomous driving. Electronic Imaging, 2017(19), 70–76.
Smith, A. B., and Katz, R. W. (2013). US billion-dollar weather and climate disasters: data sources, trends, accuracy and biases. Natural hazards, 67(2), 387–410.
The city of Detroit water and sewerage department. (2018). Stormwater Management Design Manual. Retrieved form https://detroitmi.gov/sites/detroitmi.localhost/files/2018-11/Stormwater%20Management%20Design%20Manual%202018-07-26.pdf.
Yin, Z., Zahedi, L., Leon, A. S., Amini, M. H., and Bian, L. A Machine Learning Framework for Overflow Prediction in Combined Sewer Systems. In World Environmental and Water Resources Congress 2022 (pp. 194–205).
Zhao, G., Pang, B., Xu, Z., Peng, D., and Xu, L. “Assessment of urban flood susceptibility using semi-supervised machine learning model.” Science of The Total Environment 659 (2019): 940–949.
Information & Authors
Information
Published In
World Environmental and Water Resources Congress 2023
Pages: 711 - 722
History
Published online: May 18, 2023
ASCE Technical Topics:
- Combined sewers
- Engineering fundamentals
- Environmental engineering
- Flow (fluid dynamics)
- Fluid dynamics
- Fluid mechanics
- Hydrologic engineering
- Hydrology
- Industrial wastes
- Infrastructure
- Lifeline systems
- Model accuracy
- Models (by type)
- Numerical models
- Overflow
- Pollutants
- Runoff
- Sewers
- Solid wastes
- Stormwater management
- Wastes
- Water and water resources
- Water treatment
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.