Comparison of Imputation Methods for End-User Demands in Water Distribution Systems
Publication: Journal of Water Resources Planning and Management
Volume 147, Issue 12
Abstract
This study examines the impact of advanced metering infrastructure (AMI) end-user demand metering failure on water distribution system (WDS) operation and management. To address this issue, our first step is to develop a burst detection algorithm that compares total end-user demands with system inflow rates. Western Electric Company (WEC) rules are applied to test for anomalies in the time series of normalized differences between supply and withdrawal. Then, hydraulic model prediction and burst detection performance are evaluated for fully reporting and missing AMI demand conditions using synthetically generated end-user demands for a network in Tucson, Arizona. Three imputation methods [zero, historical mean (HM), and distribution sampling (DS) method] are applied to replace missing AMI data and are compared for a range of missing data percentages. Based on the numerical experimental results, HM imputation method is the most useful tool to replace missing WDS AMI data. This scheme resulted in the lowest hydraulic model prediction errors and low false-alarm rates while maintaining high burst detection probability. However, more false alarms are raised as the percentage of missing data increases. To solve the problem, the guidelines for optimal WEC rule application are identified for a range of missing demand levels.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All of the data, models, and code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This material is based in part upon the work supported by the National Science Foundation (NSF) under Grant No. 1762862. Any opinions, finding, and conclusions or recommendations expressed in this material are those of authors and do not necessarily reflect the views of the NSF.
References
Ahn, J., and D. Jung. 2019. “Hybrid statistical process control method for water distribution pipe burst detection.” J. Water Resour. Plann. Manage. 145 (9): 06019008. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001104.
Aksela, K., and M. Aksela. 2011. “Demand estimation with automated meter reading in a distribution network.” J. Water Resour. Plann. Manage. 137 (5): 456–467. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000131.
Anele, A., Y. Hamam, A. Abu-Mahfouz, and E. Todini. 2017. “Overview, comparative assessment and recommendations of forecasting models for short-term water demand prediction.” Water 9 (11): 887. https://doi.org/10.3390/w9110887.
Aydilek, I. B., and A. Arslan. 2013. “A hybrid method for imputation of missing values using optimized fuzzy c-means with support vector regression and a genetic algorithm.” Inf. Sci. 233 (Jun): 25–35. https://doi.org/10.1016/j.ins.2013.01.021.
Bakker, M., J. H. G. Vreeburg, M. Van de Roer, and L. C. Rietveld. 2014. “Heuristic burst detection method using flow and pressure measurements.” J. Hydroinf. 16 (5): 1194–1209. https://doi.org/10.2166/hydro.2014.120.
Barrela, R., C. Amado, D. Loureiro, and A. Mamade. 2017. “Data reconstruction of flow time series in water distribution systems–A new method that accommodates multiple seasonality.” J. Hydroinf. 19 (2): 238–250. https://doi.org/10.2166/hydro.2016.192.
Blokker, E. J. M. 2010. “Stochastic water demand modelling for a better understanding of hydraulics in water distribution networks.” Ph.D. thesis, Delft Univ. of Technology. https://www.researchgate.net/publication/46395035_Stochastic_water_demand_modelling_for_a_better_understanding_of_hydraulics_in_water_distribution_networks.
Blokker, E. J. M., J. H. G. Vreeburg, and J. C. Van Dijk. 2010. “Simulating residential water demand with a stochastic end-use model.” J. Water Resour. Plann. Manage. 136 (1): 19–26. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000002.
Bragalli, C., M. Fortini, and E. Todini. 2016. “Enhancing knowledge in water distribution networks via data assimilation.” Water Resour. Manage. 30 (11): 3689–3706. https://doi.org/10.1007/s11269-016-1372-0.
Bragalli, C., M. Neri, and E. Toth. 2019. “Effectiveness of smart meter-based urban water loss assessment in a real network with synchronous and incomplete readings.” Environ. Modell. Software 112 (Feb): 128–142. https://doi.org/10.1016/j.envsoft.2018.10.010.
Buchberger, S. G., and G. J. Wells. 1996. “Intensity, duration, and frequency of residential water demands.” J. Water Resour. Plann. Manage. 122 (1): 11–19. https://doi.org/10.1061/(ASCE)0733-9496(1996)122:1(11).
City of Madison. 2019. “Project metering system.” Accessed September 23, 2019. https://www.cityofmadison.com/water/sustainability/project-h2o-metering-system.
Cominola, A., M. Giuliani, A. Castelletti, D. E. Rosenberg, and A. M. Abdallah. 2018. “Implications of data sampling resolution on water use simulation, end-use disaggregation, and demand management.” Environ. Modell. Software 102 (Apr): 199–212. https://doi.org/10.1016/j.envsoft.2017.11.022.
Creaco, E., M. Blokker, and S. Buchberger. 2017. “Models for generating household water demand pulses: Literature review and comparison.” J. Water Resour. Plann. Manage. 143 (6): 04017013. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000763.
Di Nardo, A., et al. 2015. “New Perspectives for smart water network monitoring, partitioning and protection with innovative on-line measuring sensors.” In Proc., 36th Int. Association for Hydro-Environment Engineering and Research World Congress. Beijing: International Association for Hydro-Environment Engineering and Research. https://www.iahr.org/library/world?pid=295.
Fiorillo, D., G. Galuppini, E. Creaco, F. De Paola, and M. Giugni. 2020. “Identification of influential user locations for smart meter installation to reconstruct the urban demand pattern.” J. Water Resour. Plann. Manage. 146 (8): 04020070. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001269.
Garcia-Hernandez, J. 2015. “Recent progress in the implementation of AMI projects: Standards and communications technologies.” In Proc., 2015 Int. Conf. on Mechatronics, Electronics and Automotive Engineering, 251–256. New York: IEEE.
Garrick, D. 2017. “San Diego switching to conservation-friendly ‘smart’ water meters.” Accessed August 26, 2019. https://www.sandiegouniontribune.com/news/politics/sd-me-smart-meter-20170706-story.html.
Guo, G., S. Liu, Y. Wu, J. Li, R. Zhou, and X. Zhu. 2018. “Short-term water demand forecast based on deep learning method.” J. Water Resour. Plann. Manage. 144 (12): 04018076. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000992.
Hagos, M., D. Jung, and K. E. Lansey. 2016. “Optimal meter placement for pipe burst detection in water distribution systems.” J. Hydroinf. 18 (4): 741–756. https://doi.org/10.2166/hydro.2016.170.
Horn, J. 2018. “City of San Diego could install 250,000 smart water meters.” Accessed September 23, 2019. https://www.10news.com/news/local-news/city-of-san-diego-could-install-250-000-smart-water-meters.
Huang, Y., F. Zheng, Z. Kapelan, D. Savic, H. F. Duan, and Q. Zhang. 2020. “Efficient leak localization in water distribution systems using multistage optimal valve operations and smart demand metering.” Water Resour. Res. 56 (10): e2020WR028285. https://doi.org/10.1029/2020WR028285.
Hwang, H., and K. Lansey. 2021. “Isolation valve impact on failure severity and risk analysis.” J. Water Resour. Plann. Manage. 147 (3): 04020110. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001320.
Jung, D., D. Kang, J. Liu, and K. Lansey. 2015. “Improving the rapidity of responses to pipe burst in water distribution systems: A comparison of statistical process control methods.” J. Hydroinf. 17 (2): 307–328. https://doi.org/10.2166/hydro.2014.101.
Jung, D., and J. H. Kim. 2018. “State estimation network design for water distribution systems.” J. Water Resour. Plann. Manage. 144 (1): 06017006. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000862.
Jung, D., and K. Lansey. 2015. “Water distribution system burst detection using a nonlinear Kalman filter.” J. Water Resour. Plann. Manage. 141 (5): 04014070. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000464.
Kabir, G., S. Tesfamariam, J. Hemsing, and R. Sadiq. 2020. “Handling incomplete and missing data in water network database using imputation methods.” Sustainable Resilient Infrastruct. 5 (6): 365–377. https://doi.org/10.1080/23789689.2019.1600960.
Kang, D., and K. Lansey. 2009. “Real-time demand estimation and confidence limit analysis for water distribution systems.” J. Hydraul. Eng. 135 (10): 825–837. https://doi.org/10.1061/(ASCE)HY.1943-7900.0000086.
Kang, D., and K. Lansey. 2010. “Optimal meter placement for water distribution system state estimation.” J. Water Resour. Plann. Manage. 136 (3): 337–347. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000037.
Moritz, S., and T. Bartz-Beielstein. 2017. “imputeTS: Time series missing value imputation in R.” R J. 9 (1): 207. https://doi.org/10.32614/RJ-2017-009.
Mounce, S. R., A. J. Day, A. S. Wood, A. Khan, P. D. Widdop, and J. Machell. 2002. “A neural network approach to burst detection.” Water Sci. Technol. 45 (4–5): 237–246. https://doi.org/10.2166/wst.2002.0595.
Mounce, S. R., R. B. Mounce, and J. B. Boxall. 2011. “Novelty detection for time series data analysis in water distribution systems using support vector machines.” J. Hydroinf. 13 (4): 672–686. https://doi.org/10.2166/hydro.2010.144.
Osman, M. S., A. M. Abu-Mahfouz, and P. R. Page. 2018. “A survey on data imputation techniques: Water distribution system as a use case.” IEEE Access 6 (Oct): 63279–63291. https://doi.org/10.1109/ACCESS.2018.2877269.
Peppanen, J., X. Zhang, S. Grijalva, and M. J. Reno. 2016. “Handling bad or missing smart meter data through advanced data imputation.” In Proc., Innovative Smart Grid Technol. Conf. 2016 IEEE Power & Energy Society, 1–5. New York: IEEE.
Pratama, I., A. E. Permanasari, I. Ardiyanto, and R. Indrayani. 2016. “A review of missing values handling methods on time-series data.” In Proc., Information Technology Systems and Innovation 2016 Int. Conf., 1–6. New York: IEEE.
Quevedo, J., V. Puig, G. Cembrano, J. Blanch, J. Aguilar, D. Saporta, G. Benito, M. Hedo, and A. Molina. 2010. “Validation and reconstruction of flow meter data in the Barcelona water distribution network.” Control Eng. Pract. 18 (6): 640–651. https://doi.org/10.1016/j.conengprac.2010.03.003.
Rossman, L. A. 2000. EPANET 2 user’s manual. Cincinnati: USEPA.
Rubin, D. B. 2004. Vol. 81 of Multiple imputation for nonresponse in surveys. New York: Wiley.
Stewart, R. A., R. Willis, D. Giurco, K. Panuwatwanich, and G. Capati. 2010. “Web-based knowledge management system: Linking smart metering to the future of urban water planning.” Aust. Planner 47 (2): 66–74. https://doi.org/10.1080/07293681003767769.
Taormina, R., and S. Galelli. 2018. “Deep-learning approach to the detection and localization of cyber-physical attacks on water distribution systems.” J. Water Resour. Plann. Manage. 144 (10): 04018065. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000983.
Taormina, R., S. Galelli, N. O. Tippenhauer, E. Salomons, and A. Ostfeld. 2017. “Characterizing cyber-physical attacks on water distribution systems.” J. Water Resour. Plann. Manage. 143 (5): 04017009. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000749.
Tucson Water. 2020. “Tucson water conservation program FY 2018-2019 annual report.” Accessed June 24, 2020. https://www.tucsonaz.gov/water.
United States Census Bureau. 2020. “Population and housing unit estimates.” Accessed June 24, 2020. https://www.census.gov/quickfacts/fact/table/tucsoncityarizona/PST045219.
Vertommen, I., R. Magini, and M. da Conceição Cunha. 2014. “Generating water demand scenarios using scaling laws.” Procedia Eng. 70 (Jan): 1697–1706. https://doi.org/10.1016/j.proeng.2014.02.187.
WaterWorld. 2014. “Smart water metering deployment completed at Madison water utility.” Accessed September 23, 2019. https://www.waterworld.com/municipal/technologies/amr-ami/article/16214938.
WaterWorld. 2020. “Report: Water AMI endpoints in Europe and North America to surpass 100 million units in 2025.” Accessed March 26, 2021. https://www.waterworld.com/technologies/amr-ami/press-release/14187046.
WEC (Western Electric Company). 1956. Statistical quality control handbook. 2nd ed. Indianapolis: AT&T Technologies.
Wei, R., J. Wang, M. Su, E. Jia, S. Chen, T. Chen, and Y. Ni. 2018. “Missing value imputation approach for mass spectrometry-based metabolomics data.” Sci. Rep. 8 (1): 1–10. https://doi.org/10.1038/s41598-017-19120-0.
Wu, Y., S. Liu, and X. Wang. 2018a. “Distance-based burst detection using multiple pressure sensors in district metering areas.” J. Water Resour. Plann. Manage. 144 (11): 06018009. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001001.
Wu, Z. Y., Y. He, and Q. Li. 2018b. “Comparing deep learning with statistical control methods for anomaly detection.” In Proc., 1st Int. WDSA/CCWI 2018 Joint Conf. Reston, VA: ASCE. https://ojs.library.queensu.ca/index.php/wdsa-ccw/article/view/12495.
Information & Authors
Information
Published In
Journal of Water Resources Planning and Management
Volume 147 • Issue 12 • December 2021
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Nov 2, 2020
Accepted: Aug 6, 2021
Published online: Sep 24, 2021
Published in print: Dec 1, 2021
Discussion open until: Feb 24, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Sanghoon Jun, Kevin E. Lansey, Comparison of AMI and SCADA Systems for Leak Detection and Localization in Water Distribution Networks, Journal of Water Resources Planning and Management, 10.1061/JWRMD5.WRENG-5953, 149, 11, (2023).
- Sanghoon Jun, Kevin E. Lansey, Linear Programming Models for Leak Detection and Localization in Water Distribution Networks, Journal of Water Resources Planning and Management, 10.1061/JWRMD5.WRENG-5720, 149, 5, (2023).
- Azar Niknam, Hasan Khademi Zare, Hassan Hosseininasab, Ali Mostafaeipour, A hybrid approach combining the multi-dimensional time series k-means algorithm and long short-term memory networks to predict the monthly water demand according to the uncertainty in the dataset, Earth Science Informatics, 10.1007/s12145-023-00976-y, (2023).
- Sanghoon Jun, Young Hwan Choi, Data Generation Approaches to Detect Abnormal Conditions in Water Distribution Systems, Journal of the Korean Society of Hazard Mitigation, 10.9798/KOSHAM.2022.22.2.69, 22, 2, (69-79), (2022).
- Taewook Kim, Donghwi Jung, Do Guen Yoo, Seunghyeok Hong, Sanghoon Jun, Joong Hoon Kim, Development of a Water Quality Event Detection and Diagnosis Framework in Drinking Water Distribution Systems with Structured and Unstructured Data Integration, Energies, 10.3390/en15249300, 15, 24, (9300), (2022).
- Ariele Zanfei, Andrea Menapace, Bruno Melo Brentan, Maurizio Righetti, How Does Missing Data Imputation Affect the Forecasting of Urban Water Demand?, Journal of Water Resources Planning and Management, 10.1061/(ASCE)WR.1943-5452.0001624, 148, 11, (2022).
- Zukang Hu, Wenlong Chen, Helong Wang, Pei Tian, Dingtao Shen, Integrated data-driven framework for anomaly detection and early warning in water distribution system, Journal of Cleaner Production, 10.1016/j.jclepro.2022.133977, 373, (133977), (2022).
- Sehyeong Kim, Sanghoon Jun, Donghwi Jung, Ensemble CNN Model for Effective Pipe Burst Detection in Water Distribution Systems, Water Resources Management, 10.1007/s11269-022-03291-1, 36, 13, (5049-5061), (2022).