Abstract
This paper describes a public domain semiautomated data screening and quality assurance processing tool useful for both discrete and continuous data. This application serves both a field research project assessing salinity and methylmercury drainage load control options and the requirements of a public domain watershed-based water quality forecasting model used to provide decision support for short-term salinity management in the San Joaquin River Basin of California. The US Army Corps of Engineers HEC-DSSVue data management and visualization platform was enhanced in the former application with Python scripts that performed the basic data quality assurance functions. Analogous tools were developed for use on the HOSTGATOR cloud server using the same Python scripts and algorithms to migrate quality-controlled data directly to a server for use by the Watershed Analysis Risk Management Framework (WARMF) water quality forecasting model. The Python scripts for both applications are easily adaptable by other potential users who do not currently have the resources to implement an enterprise-level hydrological data management and real-time quality assurance software system.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
The code generated and used during the study is available in a repository online in accordance with funder data retention policies and the objectives of the study.
Reproducible Results
Two files have been placed in the Github repository to provide public access to the work described in this paper and for the purposes of reproducibility:
1.
A readme file showing dashboard interface in HEC-DSSVue for performing real-time data quality screening and error corrections on monitoring stations data. (https://github.com/nwm-54/hecdss/blob/master/README.md)
2.
Annotated Python code for performing data quality control checks including data interpolation, removal of data spikes and data correction. The Python code is identical for both the HEC-DSSVue and HOSTGATOR CRON job applications. (https://github.com/nwm-54/hecdss/blob/master/dashboard.py)
Acknowledgments
The authors acknowledge initial support from the US Department of Energy Office of Science, Science Undergraduate Laboratory Internship (SULI) program at Lawrence Berkeley National Laboratory that employed intern Vi Tran during summer 2021. This work was continued by the Delta Stewardship Council-Delta Science Fellows Program (support for Stefanie Helmrich) and the Delta Science Program in partnership with the California Department of Fish and Wildlife (Contract No. 18208). We thank T. Ude and S. Segura (UC Merced) for assistance with data management and algorithm testing. Grassland Water District (Ric Ortega and Shawn Carmo) and Proposition 84 funding through the California Department of Water Resources provided modeling support and data from the District’s sensor network. The California Department of Fish and Wildlife refuge manager Shawn Allen hosted the field experimentation and provided assistance in the acquisition of essential real-time flow and salinity data.
References
Bhardwaj, J., K. K. Gupta, and R. Gupta. 2015. “A review of emerging trends on water quality measurement sensors.” In Proc., Int. Conf. on Technologies for Sustainable Development (ICTSD), 1–6. New York: IEEE.
Copp, J., E. Belia, C. Hubner, M. Thron, P. A. Vanrolleghem, and L. Rieger. 2010. “Towards the automation of water quality monitoring networks.” In Proc., Automation Science and Engineering (CASE), 491–496. New York: IEEE.
Demetillo, A. T., M. V. Japitana, and E. B. Taboada. 2019. “A system for monitoring water quality in a large aquatic area using wireless sensor network technology.” Sustainable Environ. Res. 29 (1): 12. https://doi.org/10.1186/s42834-019-0009-4.
Faustine, A., A. N. Mvuma, H. J. Mongi, M. C. Gabriel, A. J. Tenge, and S. B. Kucel. 2014. “Wireless sensor networks for water quality monitoring and control within Lake Victoria Basin: Prototype development.” Wireless Sensor Networks 2014 (6): 281–290.
Herr, J., L. H. Z. Weintraub, and C. W. Chen. 2001. User’s gide to WARMF: Documentation of graphical user interface. Topical Report. Palo Alto, CA: EPRI.
Horsburgh, J. S., S. L. Reeder, A. S. Jones, and J. Meline. 2015. “Open-source software for visualization and quality control of continuous hydrologic and water quality sensor data.” Environ. Modell. Software 70 (Aug): 32–44. https://doi.org/10.1016/j.envsoft.2015.04.002.
Kotamaki, N., S. Thessler, J. Koskiaho, A. O. Hannukkala, H. Huitu, and T. Huttula. 2009. “Wireless in-situ sensor network for agriculture and water monitoring on a river basin scale in southern Finland: Evaluation from a data user’s perspective.” Sensors 9 (Jun): 2862–2883.
Moatar, F., J. Miquel, and A. Poirel. 2001. “A quality-control method for physical and chemical monitoring data.” J. Hydrol. 252 (1–4): 25–36. https://doi.org/10.1016/S0022-1694(01)00439-5.
Plana, Q., K. Alferes, K. Fuks, T. Kraft, T. Maruéjouls, E. Torfs, and P. A. Vanrolleghem. 2019. “Towards a water quality database for raw and validated data with emphasis on structured metadata.” Water Qual. Res. J. 54 (1): 1–9. https://doi.org/10.2166/wqrj.2018.013.
Quinn, N. W. T. 2006. “Bottom-up, decision support system development: A wetland salinity management application in California’s San Joaquin Valley.” In Summit on environmental modeling and software 3rd biennial meeting of the international environmental modelling and software society. Provo, UT: Brigham Young Univ.
Quinn, N. W. T., and W. M. Hanna. 2002. “Real-time adaptive management of seasonal wetlands to improve water quality in the San Joaquin River.” Adv. Environ. Res. 5 (4): 309–317. https://doi.org/10.1016/S1093-0191(01)00081-8.
Quinn, N. W. T., and W. M. Hanna. 2003. “A decision support system for adaptive real-time management of seasonal wetlands in California.” Environ. Modell. Softw. 18 (6): 503–511.
Randhawa, S., S. S. Sandha, and B. Srivastava. 2016. “A multi-sensor process for in-situ monitoring of water pollution in rivers or lakes for high-resolution quantitative and qualitative water quality data.” In Proc., IEEE Intl Conf. on Computational Science and Engineering (CSE) and IEEE Int. Conf. on Embedded and Ubiquitous Computing (EUC) and 15th Intl Symp. on Distributed Computing and Applications for Business Engineering (DCABES). New York: IEEE. https://doi.org/10.1109/CSE-EUC-DCABES.2016.171.
Rasin, Z., and M. R. Abdullah. 2009. “Water quality monitoring system using Zigbee based wireless sensor network.” Int. J. Eng. Technol. 9 (10): 14–18.
Talagala, P. D., R. J. Hyndman, C. Leigh, K. Mengersen, and K. Smith-Miles. 2019. “A feature-based procedure for detecting technical outliers in water-quality data from in situ sensors.” Water Resour. Res. 55 (11): 8547–8568. https://doi.org/10.1029/2019WR024906.
USCOE (US Army Corps of Engineers). 2009. User’s manual HEC-DSSVue HEC data storage system visual utility engine. Davis, CA: USCOE.
Wagner, R. J., R. W. Boulger Jr., C. J. Oblinger, and B. A. Smith. 2006. Guidelines and standard procedures for continuous water-quality monitors: Station operation, record computation, and data reporting. Washington, DC: USGS.
Information & Authors
Information
Published In
Journal of Water Resources Planning and Management
Volume 149 • Issue 9 • September 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Jan 27, 2022
Accepted: May 2, 2023
Published online: Jul 12, 2023
Published in print: Sep 1, 2023
Discussion open until: Dec 12, 2023
ASCE Technical Topics:
- Business management
- Computer languages
- Computer programming
- Computer software
- Computing in civil engineering
- Decision making
- Decision support systems
- Engineering fundamentals
- Environmental engineering
- Information management
- Management methods
- Practice and Profession
- Quality control
- River engineering
- River systems
- Salt water
- Water (by type)
- Water and water resources
- Water management
- Water quality
- Water treatment
- Watersheds
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- James H. Stagge, David E. Rosenberg, Anthony M. Castronova, Avi Ostfeld, Amber Spackman Jones, Journal of Water Resources Planning and Management’s Reproducibility Review Program: Accomplishments, Lessons, and Next Steps, Journal of Water Resources Planning and Management, 10.1061/JWRMD5.WRENG-6559, 150, 8, (2024).