Noise Removal for the Steady-State Pressure Measurements Based on Domain Knowledge of Water Distribution Systems
Publication: Journal of Water Resources Planning and Management
Volume 150, Issue 3
Abstract
The pressure sensors have been widely used to capture the steady-state pressure data for the management of water distribution systems (WDSs). The background noise pollution in WDSs can lead to signal degradation, reducing the reliability of sensor data. Although numerous algorithms have been developed for noise removal, their optimal parameter selection is problem-dependent. The steady-state pressure sensor data are often linearly correlated, and this correlation is related to hydraulic distance. This is a key feature of WDS pressure sensor data that should be considered in noise-removal algorithms. Based on this domain knowledge, two noise-removal metrics are proposed to evaluate the performance of the noise-removal algorithm and aid in selecting the optimal parameters. In addition, four widely used noise-removal algorithms have been reviewed and used for removing noise from the pressure sensor data. The results demonstrate that the noise-removal algorithm can efficiently remove potential random noise and enhance the reliability of the sensor data.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request. All the field data in Case Study 2, including pressures, reservoir outflows, hydraulic model, and so on, are confidential and cannot be provided without third-party agreement.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (No. 52070165) and the National Natural Science Foundation of China (No. 52200119).
References
Abokifa, A. A., K. Haddad, C. Lo, and P. Biswas. 2019. “Real-time identification of cyber-physical attacks on water distribution systems via machine learning–based anomaly detection techniques.” J. Water Resour. Plann. Manage. 145 (1): 04018089 https://doi.org/10.1061/(ASCE)WR.1943-5452.0001023.
Alexandrov, T., S. Bianconcini, E. B. Dagum, P. Maass, and T. S. McElroy. 2012. “A review of some modern approaches to the problem of trend extraction.” Econ. Rev. 31 (6): 593–624. https://doi.org/10.1080/07474938.2011.608032.
Aubert, G., and J.-F. Aujol. 2008. “A variational approach to removing multiplicative noise.” SIAM J. Appl. Math. 68 (4): 925–946. https://doi.org/10.1137/060671814.
Badri, L. 2010. “Development of neural networks for noise reduction.” Int. Arab J. Inf. Technol. 7 (3): 289–294.
Bornoiu, I.-V., and O. Grigore. 2013. “A study about feature extraction for stress detection.” In Proc., 2013 8th Int. Symp. on Advanced Topics in Electrical Engineering (ATEE), 1–4. New York: IEEE.
Brockwell, P. J., and R. A. Davis. 2016. Introduction to time series and forecasting. Cham, Switzerland: Springer.
Canazza, S., G. De Poli, and G. A. Mian. 2010. “Restoration of audio documents by means of extended Kalman filter.” IEEE Trans. Audio Speech Lang. Process. 18 (6): 1107–1115. https://doi.org/10.1109/TASL.2009.2030005.
Christov, I., T. Neycheva, and R. Schmid. 2017. “Fine tuning of the dynamic low-pass filter for electromyographic noise suppression in electrocardiograms.” In Proc., 2017 Computing in Cardiology (CinC), 1–4. New York: IEEE.
Chu, S., T. Zhang, Y. Shao, T. Yu, and H. Yao. 2020. “Numerical approach for water distribution system model calibration through incorporation of multiple stochastic prior distributions.” Sci. Total Environ. 708 (Mar): 134565. https://doi.org/10.1016/j.scitotenv.2019.134565.
Do, N. C., A. R. Simpson, J. W. Deuerlein, and O. Piller. 2017. “Particle filter–based model for online estimation of demand multipliers in water distribution systems under uncertainty.” J. Water Resour. Plann. Manage. 143 (11): 04017065 https://doi.org/10.1061/(ASCE)WR.1943-5452.0000841.
Donoho, D. L., and I. M. Johnstone. 1994. “Ideal denoising in an orthonormal basis chosen from a library of bases.” C. R. Acad. Sci. Sér. I 319 (12): 1317–1322.
Dubé, A.-A., M. Duquette, M. Roy, F. Lepore, G. Duncan, and P. Rainville. 2009. “Brain activity associated with the electrodermal reactivity to acute heat pain.” NeuroImage 45 (1): 169–180. https://doi.org/10.1016/j.neuroimage.2008.10.024.
Eliades, D. G., M. Kyriakou, S. Vrachimis, and M. M. Polycarpou. 2016. “EPANET-MATLAB toolkit: An open-source software for interfacing EPANET with MATLAB.” In Proc., 14th Int. Conf. on Computing and Control for the Water Industry (CCWI), 8. Amsterdam, Netherlands: Zenodo. https://doi.org/10.5281/zenodo.831492.
Ferrante, M., B. Brunone, and S. Meniconi. 2007. “Wavelets for the analysis of transient pressure signals for leak detection.” J. Hydraul. Eng. 133 (11): 1274–1282. https://doi.org/10.1061/(ASCE)0733-9429(2007)133:11(1274).
Gao, S., D. McLean, J. Lai, C. Micou, and A. Nathan. 2016. “Reduction of noise spikes in touch screen systems by low pass spatial filtering.” J. Disp. Technol. 12 (9): 957–963. https://doi.org/10.1109/JDT.2016.2550519.
Ghibaudo, J., M. Aucejo, and O. De Smet. 2022. “A sparse adaptive Bayesian filter for input estimation problems.” Mech. Syst. Signal Process. 180 (Nov): 109416. https://doi.org/10.1016/j.ymssp.2022.109416.
Hoti, R. B. K., and S. Khattak. 2014. “Automated noise removal and feature extraction in ECGs.” Arabian J. Sci. Eng. 39 (3): 1937–1948. https://doi.org/10.1007/s13369-013-0801-0.
Hutton, C. J., Z. Kapelan, L. Vamvakeridou-Lyroudia, and D. A. Savić. 2014. “Dealing with uncertainty in water distribution system models: A framework for real-time modeling and data assimilation.” J. Water Resour. Plann. Manage. 140 (2): 169–183. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000325.
Kaufman, L., and P. J. Rousseeuw. 2005. “Finding groups in data: An introduction to cluster analysis.” In Wiley series in probability and mathematical statistics. Hoboken, NJ: Wiley.
Kim, S., and N. Park. 2012. “Noise filtering for transient data in a pipeline.” In Proc., Bridging the Gap: Meeting the World’s Water and Environmental Resources Challenges, 1–9. Reston, VA: ASCE. https://doi.org/10.1061/40569(2001)299.
Kopsinis, Y., and S. McLaughlin. 2009. “Development of EMD-based denoising methods inspired by wavelet thresholding.” IEEE Trans. Signal Process. 57 (4): 1351–1362. https://doi.org/10.1109/TSP.2009.2013885.
Lee, G., B. Choi, H. Jebelli, C. R. Ahn, and S. Lee. 2019. “Reference signal-based method to remove respiration noise in electrodermal activity (EDA) collected from the field.” In Proc., ASCE Int. Conf. on Computing in Civil Engineering, 17–25. Reston, VA: ASCE. https://doi.org/10.1061/9780784482438.003.
Lee, G., B. Choi, H. Jebelli, C. R. Ahn, and S. Lee. 2020. “Noise reference signal–based denoising method for EDA collected by multimodal biosensor wearable in the field.” J. Comput. Civ. Eng. 34 (6): 04020044 https://doi.org/10.1061/(ASCE)CP.1943-5487.0000927.
Lentka, Ł., and J. Smulko. 2019. “Methods of trend removal in electrochemical noise data–Overview.” Measurement 131 (Jan): 569–581. https://doi.org/10.1016/j.measurement.2018.08.023.
Li, L., J. Yan, Q. Wen, Y. Jin, and X. Yang. 2022. “Learning robust deep state space for unsupervised anomaly detection in contaminated time-series.” IEEE Trans. Knowl. Data Eng. 35 (6): 6058–6072. https://doi.org/10.1109/TKDE.2022.3171562.
Li, Z., and C. C. Chang. 2013. “Adaptive quantification of noise variance using subspace technique.” J. Eng. Mech. 139 (4): 469–478. https://doi.org/10.1061/(ASCE)EM.1943-7889.0000499.
Liu, C., R. Szeliski, S. Bing Kang, C. L. Zitnick, and W. T. Freeman. 2008. “Automatic estimation and removal of noise from a single image.” IEEE Trans. Pattern Anal. Mach. Intell. 30 (2): 299–314. https://doi.org/10.1109/TPAMI.2007.1176.
Mahajan, R., and D. Padha. 2020. “Multiresolution wavelet transform based anisotropic diffusion for removing speckle noise in a real-time vision-based database.” Int. J. Multimedia Data Eng. Manage. 11 (1): 1–14. https://doi.org/10.4018/IJMDEM.2020010101.
Mallat, S. G. 1989. “A theory for multiresolution signal decomposition: The wavelet representation.” IEEE Trans. Pattern Anal. Mach. Intell. 11 (7): 674–693. https://doi.org/10.1109/34.192463.
Mori, N., T. Suzuki, and S. Kakuno. 2007. “Noise of acoustic Doppler velocimeter data in bubbly flows.” J. Eng. Mech. 133 (1): 122–125. https://doi.org/10.1061/(ASCE)0733-9399(2007)133:1(122).
Niedzwiecki, M., and K. Cisowski. 1996. “Adaptive scheme for elimination of broadband noise and impulsive disturbances from AR and ARMA signals.” IEEE Trans. Signal Process. 44 (3): 528–537. https://doi.org/10.1109/78.489026.
Niedźwiecki, M., and M. Ciołek. 2015. “Localization of impulsive disturbances in audio signals using template matching.” Digital Signal Process. 46 (Nov): 253–265. https://doi.org/10.1016/j.dsp.2015.07.009.
Niedźwiecki, M., and M. Ciołek. 2019. “Generalized Savitzky–Golay filters for identification of nonstationary systems.” Automatica 108 (Oct): 108477. https://doi.org/10.1016/j.automatica.2019.06.029.
Puust, R., Z. Kapelan, D. A. Savic, and T. Koppel. 2010. “A review of methods for leakage management in pipe networks.” Urban Water J. 7 (1): 25–45. https://doi.org/10.1080/15730621003610878.
Rahman, M. A., M. A. Rashid, and M. Ahmad. 2019. “Selecting the optimal conditions of Savitzky–Golay filter for fNIRS signal.” Biocybern. Biomed. Eng. 39 (3): 624–637. https://doi.org/10.1016/j.bbe.2019.06.004.
Rajakumar, A. G., M. S. Mohan Kumar, B. Amrutur, and Z. Kapelan. 2019. “Real-time water quality modeling with ensemble Kalman filter for state and parameter estimation in water distribution networks.” J. Water Resour. Plann. Manage. 145 (11): 04019049 https://doi.org/10.1061/(ASCE)WR.1943-5452.0001118.
Rossman, L., H. Woo, M. Tryby, F. Shang, R. Janke, and T. Haxton. 2020. “EPANET 2.2 user’s manual, water infrastructure division.” In Center for Environmental Solutions and Emergency Response. Cincinnati: Office of Research and Development, USEPA.
Rouis, M., S. Sbaa, and N. E. Benhassine. 2020. “The effectiveness of the choice of criteria on the stationary and non-stationary noise removal in the phonocardiogram (PCG) signal using discrete wavelet transform.” Biomed. Technol. 65 (3): 353–366. https://doi.org/10.1515/bmt-2019-0197.
Shao, Y., S. Chu, T. Zhang, Y. J. Yang, and T. Yu. 2019. “Real-time water distribution system hydraulic modeling using prior demand information by formal Bayesian approach.” J. Water Resour. Plann. Manage. 145 (12): 04019059 https://doi.org/10.1061/(ASCE)WR.1943-5452.0001137.
Shrestha, P., and A. H. Behzadan. 2018. “Chaos theory–inspired evolutionary method to refine imperfect sensor data for data-driven construction simulation.” J. Constr. Eng. Manage. 144 (3): 04018001 https://doi.org/10.1061/(ASCE)CO.1943-7862.0001441.
So, S., and K. K. Paliwal. 2011. “Suppressing the influence of additive noise on the Kalman gain for low residual noise speech enhancement.” Speech Commun. 53 (3): 355–378. https://doi.org/10.1016/j.specom.2010.10.006.
Steidl, G., and T. Teuber. 2010. “Removing multiplicative noise by Douglas-Rachford splitting methods.” J. Math. Imaging Vis. 36 (2): 168–184. https://doi.org/10.1007/s10851-009-0179-5.
Tan, Y. J., S. Bailey, and B. Kinsella. 1996. “The monitoring of the formation and destruction of corrosion inhibitor films using electrochemical noise analysis (ENA).” Corros. Sci. 38 (10): 1681–1695. https://doi.org/10.1016/S0010-938X(96)00061-3.
Wan, H., X. Ma, and X. Li. 2018. “Variational Bayesian learning for removal of sparse impulsive noise from speech signals.” Digital Signal Process. 73 (Feb): 106–116. https://doi.org/10.1016/j.dsp.2017.11.007.
Wang, Z., J. Guan, J. Sun, J. Yu, and J. Liu. 2012. “Expressway data de-noised approach based on wavelet analysis.” In Proc., ICCTP 2009: Critical Issues In Transportation Systems Planning, Development, and Management, 1–7. Reston, VA: ASCE. https://doi.org/10.1061/41064(358)28.
Wu, Y., S. Liu, K. Smith, and X. Wang. 2018. “Using correlation between data from multiple monitoring sensors to detect bursts in water distribution systems.” J. Water Resour. Plann. Manage. 144 (2): 04017084 https://doi.org/10.1061/(ASCE)WR.1943-5452.0000870.
Yang, H. 2013. “Solving problems of imperfect data streams by incremental decision trees.” JETWI 5 (3): 322–331. https://doi.org/10.4304/jetwi.5.3.322-331.
Zhang, Q., F. Zheng, H.-F. Duan, Y. Jia, T. Zhang, and X. Guo. 2018. “Efficient numerical approach for simultaneous calibration of pipe roughness coefficients and nodal demands for water distribution systems.” J. Water Resour. Plann. Manage. 144 (10): 04018063 https://doi.org/10.1061/(ASCE)WR.1943-5452.0000986.
Zhang, Y., and J. Li. 2006. “Wavelet-based vibration sensor data compression technique for civil infrastructure condition monitoring.” J. Comput. Civ. Eng. 20 (6): 390–399. https://doi.org/10.1061/(ASCE)0887-3801(2006)20:6(390).
Zhou, C., and Y. Zhang. 2012. “Particle filter based noise removal method for acoustic emission signals.” Mech. Syst. Signal Process. 28 (Apr): 63–77. https://doi.org/10.1016/j.ymssp.2011.08.004.
Information & Authors
Information
Published In
Journal of Water Resources Planning and Management
Volume 150 • Issue 3 • March 2024
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Apr 30, 2023
Accepted: Oct 12, 2023
Published online: Dec 21, 2023
Published in print: Mar 1, 2024
Discussion open until: May 21, 2024
ASCE Technical Topics:
- Algorithms
- Continuum mechanics
- Dynamics (solid mechanics)
- Engineering fundamentals
- Engineering mechanics
- Environmental engineering
- Equipment and machinery
- Infrastructure
- Material mechanics
- Material properties
- Materials engineering
- Mathematics
- Mechanical properties
- Noise pollution
- Pipeline systems
- Pipes
- Pollution
- Pressure (type)
- Pressure distribution
- Pressure pipes
- Probe instruments
- Solid mechanics
- Steady states
- Tension
- Water and water resources
- Water management
- Water pressure
- Water supply
- Water supply systems
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.