Abstract
Data on the quality of small-scale infrastructure assets at the neighborhood level tend to be scarce, delayed, or even nonexistent, limiting the ability of local communities to better manage assets, identify risks, and improve performance. Participatory methods, common in ecology and environmental science, provide an opportunity for producing these fine data. However, very little is known about the validity and reliability of infrastructure data collected by volunteer members of the public. In this paper, we examine the validity and reliability of local drainage infrastructure monitoring data obtained from citizen scientists. The volunteer citizen scientists were high school students collecting data about the condition of stormwater infrastructure in their local community. The study took place in an environmental justice neighborhood in Houston with a long history of flooding and exposure to environmental hazards. The results suggest that with adequate planning, training, and organized community engagement efforts, the validity and reliability of data collected by citizen scientists can be comparable to physical measurements and data obtained from trained inspectors.
Get full access to this article
View all available purchase options and get full access to this article.
References
ALDOT (Alabama Department of Transportation). 2015. Level of service condition assessments data collection manual. Montgomery, AL: ALDOT.
Bird, T. J., et al. 2014. “Statistical solutions for error and bias in global citizen science datasets.” Biol. Conserv. 173: 144–154. https://doi.org/10.1016/j.biocon.2013.07.037.
Bonney, R., C. B. Cooper, J. Dickinson, S. Kelling, T. Phillips, K. V. Rosenberg, and J. Shirk. 2009. “Citizen science: A developing tool for expanding science knowledge and scientific literacy.” BioScience 59 (11): 977–984. https://doi.org/10.1525/bio.2009.59.11.9.
Bonney, R., J. L. Shirk, T. B. Phillips, A. Wiggins, H. L. Ballard, A. J. Miller-Rushing, and J. K. Parrish. 2014. “Next steps for citizen science.” Science 343 (6178): 1436–1437. https://doi.org/10.1126/science.1251554.
Buchheit, R. B., J. H. Garrett Jr., S. McNeil, and P. Chen. 2005. “Automated procedure to assess civil infrastructure data quality: Method and validation.” J. Infrastruct. Syst. 11 (3): 180–189. https://doi.org/10.1061/(ASCE)1076-0342(2005)11:3(180).
Bukenya, B., A. Ahmed, J. M. Andrade, D. S. Grigsby-Toussaint, J. Muyonga, and J. E. Andrade. 2017. “Validity and reliability of general nutrition knowledge questionnaire for adults in Uganda.” Nutrients 9 (2): 172. https://doi.org/10.3390/nu9020172.
Bullard, R. D., and B. H. Wright. 1993. “Environmental justice for all: Community perspectives on health and research needs.” Toxicol. Ind. Health 9 (5): 821–841. https://doi.org/10.1177/074823379300900508.
City of Houston. 2016. Infrastructure design manual. Houston: Dept. of Public Works and Engineering, City of Houston.
Corburn, J. 2005. Street science: Community knowledge and environmental health justice. Cambridge, MA: MIT Press.
Delaney, D., C. Sperling, C. Adams, and B. Leung. 2008. “Marine invasive species: Validation of citizen science and implications for national monitoring networks.” Biol. Invasions 10 (1): 117–128. https://doi.org/10.1007/s10530-007-9114-0.
Dickinson, J. L., J. Shirk, D. Bonter, R. Bonney, R. L. Crain, J. Martin, T. Phillips, and K. Purcell. 2012. “The current state of citizen science as a tool for ecological research and public engagement.” Front. Ecol. Environ. 10 (6): 291–297. https://doi.org/10.1890/110236.
Feinstein, A. R., and D. V. Cicchetti. 1990. “High agreement but low kappa. I: The problems of two paradoxes.” J. Clin. Epidemiol. 43 (6): 543–549. https://doi.org/10.1016/0895-4356(90)90158-L.
Feng, G. C. 2014. “Mistakes and how to avoid mistakes in using intercoder reliability indices.” Methodology 11 (1): 13–22. https://doi.org/10.1027/1614-2241/a000086.
Fleiss, J. L. 1981. Statistical methods for rates and proportions. New York: Wiley.
Gharaibeh, N. G., and D. B. Lindholm. 2014. “A condition assessment method for roadside assets.” Struct. Infrastruct. Eng. Maint. Manage. Life-Cycle Des. Perform. 10 (3): 409–418. https://doi.org/10.1080/15732479.2012.757330.
Goodchild, M. F., and L. Li. 2012. “Assuring the quality of volunteered geographic information.” Spatial Stat. 1: 110–120. https://doi.org/10.1016/j.spasta.2012.03.002.
Gwet, K. 2014. Handbook of inter-rater reliability. Gaithersburg, MD: Advanced Analytics, LLC.
Halfawy, M. R. 2008. “Integration of municipal infrastructure asset management processes: Challenges and solutions.” J. Comput. Civ. Eng. 22 (3): 216–229. https://doi.org/10.1061/(ASCE)0887-3801(2008)22:3(216).
Hendricks, M. D., M. A. Meyer, N. G. Gharaibeh, S. Van Zandtd, J. Masterson, J. A. Horney, and P. Berke. 2018. “The development of a participatory assessment technique for infrastructure: Neighborhood-level monitoring towards sustainable infrastructure systems.” Sustainable Cities Soc. 38: 265–274. https://doi.org/10.1016/j.scs.2017.12.039.
ISO. 1994. Accuracy (trueness and precision) of measurement methods and results—Part 1: General principles and definitions. BS ISO 5725-1. Geneva: ISO.
Juni, E., T. M. Adams, and D. Sokolowski. 2008. “Relating cost to condition in routine highway maintenance.” Transp. Res. Rec. 2044: 3–10. https://doi.org/10.3141/2044-01.
Kosmala, M., A. Wiggins, A. Swanson, and B. Simmons. 2016. “Assessing data quality in citizen science.” Front. Ecol. Environ. 14 (10): 551–560. https://doi.org/10.1002/fee.1436.
Krippendorff, K. 2004. Content analysis: An introduction to its methodology. 2nd ed. Thousand Oaks, CA: Sage.
Krippendorff, K. 2011. “Agreement and information in the reliability of coding.” Commun. Methods Measures 5 (2): 93–112. https://doi.org/10.1080/19312458.2011.568376.
Law, K. H., K. Smarsly, Y. Wang, M. L. Wang, J. P. Lynch, and H. Sohn. 2014. “Sensor data management technologies for infrastructure asset management.” In Sensor technologies for civil infrastructures: Applications in structural health monitoring, 3–32. Oxford, UK: Woodhead Publishing.
Molzahn, R. E., and C. B. Burke. 1986. “Assessment of drainage infrastructure for urban areas.” Tunnelling Underground Space Technol. 1 (3/4): 387–389. https://doi.org/10.1016/0886-7798(86)90022-2.
NCDOT (North Carolina Department of Transportation). 2003. Drainage users manual. Raleigh, NC: Division of Highways.
Pandya, R. E. 2012. “A framework for engaging diverse communities in citizen science in the US.” Front. Ecol. Environ. 10 (6): 314–317. https://doi.org/10.1890/120007.
Pantelias, A., G. W. Flintsch, J. W. Bryant Jr., and C. Chen. 2009. “Asset management data practices for supporting project selection decisions.” Public Works Manage. Policy 13 (3): 239–252. https://doi.org/10.1177/1087724X08327574.
Pierce, L. M., G. McGovern, and K. A. Zimmerman. 2013. Practical guide for quality management of pavement condition data collection. Washington, DC: Federal Highway Administration, US Dept. of Transportation.
Saliminejad, S., and N. G. Gharaibeh. 2013. “Impact of error in pavement condition data on output of network-level pavement management systems.” Transp. Res. Rec. 2366: 110–119. https://doi.org/10.3141/2366-13.
Schlossberg, M. A., and C. Brehm. 2009. “Participatory geographic information systems and active transportation: Collecting data and creating change.” Transp. Res. Rec. 2105: 83–91. https://doi.org/10.3141/2105-11.
Steinke, J., J. van Etten, and P. M. Zelan. 2017. “The accuracy of farmer-generated data in an agricultural citizen science methodology.” Agron. Sustaintable Dev. 37 (4): 32. https://doi.org/10.1007/s13593-017-0441-y.
Stivers, M. L., K. L. Smith, T. E. Hoerner, and A. R. Romine. 1999. Maintenance QA program implementation manual. Washington, DC: Transportation Research Board.
Taylor, D. 2014. Toxic communities: Environmental racism, industrial pollution, and residential mobility. New York: NYU Press.
UDOT (Utah Department of Transportation). 2012. Maintenance management quality assurance plus (MMQA+), inspection manual. Taylorsville, UT: UDOT.
Walker, D., L. Entine, and S. Kummer. 2000. Local road assessment and improvement drainage manual. Madison, WI: Transportation Information Center, Univ. of Wisconsin–Madison.
Watson, P. F., and A. Petrie. 2010. “Method agreement analysis: A review of correct methodology.” Theriogenology 73 (9): 1167–1179. https://doi.org/10.1016/j.theriogenology.2010.01.003.
Zhai, Z., D. Hachen, T. Kijewski-Correa, F. Shen, and G. Madey. 2012. “Citizen engineering: Methods for crowdsourcing highly trustworthy results.” In Proc., 45th Hawaii Int. Conf. on System Sciences, 3406–3415. New York: IEEE.
Information & Authors
Information
Published In
Journal of Infrastructure Systems
Volume 25 • Issue 3 • September 2019
Copyright
©2019 American Society of Civil Engineers.
History
Received: Jun 12, 2018
Accepted: Dec 18, 2018
Published online: Apr 25, 2019
Published in print: Sep 1, 2019
Discussion open until: Sep 25, 2019
ASCE Technical Topics:
- Asset management
- Business management
- Data collection
- Drainage
- Engineering fundamentals
- Engineering profession
- Financial management
- Information management
- Infrastructure
- Infrastructure vulnerability
- Irrigation engineering
- Methodology (by type)
- Practice and Profession
- Professional development
- Research methods (by type)
- Residential location
- Urban and regional development
- Urban areas
- Volunteerism
- Water and water resources
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.