Automatic Damage Segmentation Framework for Buried Sewer Pipes Based on Machine Vision: Case Study of Sewer Pipes in Zhengzhou, China
Publication: Journal of Infrastructure Systems
Volume 29, Issue 1
Abstract
Aging buried sewer pipes are at risk of rupture, corrosion, scaling, and other damages, and they may cause urban waterlogging, road collapse, environmental pollution, and other urban safety problems. Therefore, it is very important to accurately detect and segment sewer pipe damages regularly for safety evaluation. This paper proposes an intelligent damage detection method to segment and measure five types of sewer pipe damages, namely, staggered joints, deposits, corrosion, scaling, and fracture, leveraging a fine-tuned fully convolutional network (FCN) algorithm. A case study on Zhengzhou’s sewer pipes is conducted. Then, a dataset of 3,558 images is built, and the images are collected from Zhengzhou’s sewer pipes. Subsequently, the encoder of the FCN is replaced with a pretrained VGG-19, and the upsampling part has spatial and channel squeeze and excitation (SCSE) modules added. The evaluation matrix of the pixel accuracy (PA), mean pixel accuracy (MPA), mean intersection over union (MIoU), and frequency weighted intersection over union (FWIoU) of the proposed method is 91.42%, 81.44%, 73.65%, and 87.16%, respectively. Subsequently, damage areas are measured at a pixel level to quantify the pipe damage. Compared with the existing mask region-based convolutional neural networks (RCNN) method, the proposed FCN-based method shows effective performance for multiple and similar sewer pipe damage detection under complex backgrounds. Moreover, a real-time damage detection platform for sewer pipes is proposed, realizing the data uploading, data storage, data analysis, and data visualization.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or code that supports the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
The research is supported by the National Natural Science Foundation of China (51978630), National Key R & D Program of China (2022YFC3801000), Program for Science & Technology Innovation Talents in Universities of Henan Province (23HASTIT006), Postdoctoral Research Foundation of China (2022TQ0306), Key Scientific Research Projects of Higher Education in Henan Province (21A560013), Open Fund of Changjiang Institute of Survey, Lanning, Design and Research (CX2020K10).
References
Abdel-Qader, I., O. Abudayyeh, and M. Kelly. 2003. “Analysis of edge-detection techniques for crack identification in bridges.” J. Comput. Civ. Eng. 17 (4): 255–263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255).
Albatayneh, O., L. Forslof, and K. Ksaibati. 2020. “Image retraining using TensorFlow implementation of the pretrained Inception-v3 model for evaluating gravel road dust.” J. Infrastrut. Syst. 26 (2): 04020014. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000545.
Ali, R., and Y. Cha. 2019. “Subsurface damage detection of a steel bridge using deep learning and uncooled micro-bolometer.” Constr. Build. Mater. 226 (Nov): 376–387. https://doi.org/10.1016/j.conbuildmat.2019.07.293.
Assaad, R., and I. El-adaway. 2020. “Bridge infrastructure asset management system: Comparative computational machine learning approach for evaluating and predicting deck deterioration conditions.” J. Infrastrut. Syst. 26 (3): 04020032. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000572.
Badrinarayanan, V., A. Kendall, and R. Cipolla. 2017. “SegNet: A deep convolutional encoder-decoder architecture for image segmentation.” IEEE Trans. Pattern Anal. 39 (12): 2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615.
Boas, I., C. Chen, H. Wiegel, and G. He. 2020. “The role of social media-led and governmental information in China’s urban disaster risk response: The case of Xiamen.” Int. J. Disaster Risk Reduct. 51 (Dec): 101905. https://doi.org/10.1016/j.ijdrr.2020.101905.
Cantos, W., I. Juran, and S. Tinelli. 2020. “Machine-learning-based risk assessment method for leak detection and geolocation in a water distribution system.” J. Infrastruct. Syst. 26 (1): 04019039. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000517.
Cha, Y., K. You, and W. Choi. 2016. “Vision-based detection of loosened bolts using the Hough transform and support vector machines.” Autom. Constr. 71 (Part 2): 181–188. https://doi.org/10.1016/j.autcon.2016.06.008.
Cheng, J., and M. Wang. 2018. “Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques.” Autom. Constr. 95 (Nov): 155–171. https://doi.org/10.1016/j.autcon.2018.08.006.
Damhuji, R., A. Ahmed, R. Ramdan, A. Kharudin, and F. Moneer. 2017. “An eddy current testing platform system for pipe defect inspection based on an optimized eddy current technique probe design.” Sensors 17 (3): 579. https://doi.org/10.3390/s17030579.
Duran, O., K. Althoefer, and L. Seneviratne. 2007. “Automated pipe defect detection and categorization using camera/laser-based profiler and artificial neural network.” IEEE Trans. Autom. Sci. Eng. 4 (1): 118–126. https://doi.org/10.1109/TASE.2006.873225.
Fujita, Y., Y. Mitani, and Y. Hamamoto. 2006. “A method for crack detection on a concrete structure.” In Proc., IEEE 18th Int. Conf. on Pattern Recognition, 513–516. Hong Kong: Hong Kong Baptist Univ.
Garcia-Garcia, A., S. Orts-Escolano, S. Oprea, V. Villena-Martinez, and J. Garcia-Rodriguez. 2017. “A review on deep learning techniques applied to semantic segmentation.” Preprint, submitted April 27, 2017. http://arxiv.org/abs/1704.06857.
German, S., I. Brilakis, and R. DesRoches. 2012. “Rapid entropy-based detection and properties measurement of concrete spalling with machine vision for post-earthquake safety assessments.” Adv. Eng. Inform. 26 (4): 846–858. https://doi.org/10.1016/j.aei.2012.06.005.
Girshick, R. 2015. “Fast R-CNN.” In Proc., IEEE Int. Conf. on Computer Vision, 1440–1448. New York: IEEE.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2015. “Region-based convolutional networks for accurate object detection and segmentation.” IEEE Trans. Pattern Anal. 38 (1): 142–158. https://doi.org/10.1109/TPAMI.2015.2437384.
Hanif, M., Z. Ibrahim, K. Ghaedi, H. Hashim, and A. Javanmardi. 2018. “Damage assessment of reinforced concrete structures using a model-based nonlinear approach—A comprehensive review.” Constr. Build. Mater. 192 (Dec): 846–865. https://doi.org/10.1016/j.conbuildmat.2018.10.115.
He, K., and G. Gkioxari. 2017. “Mask R-CNN.” In Proc., IEEE Int. Conf on Computer Vision, 2961–2969. New York: IEEE.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Iyer, S., and S. Sinha. 2005. “A robust approach for automatic detection and segmentation of cracks in underground pipeline images.” Image Vision Comput. 23 (10): 921–933. https://doi.org/10.1016/j.imavis.2005.05.017.
Iyer, S., and S. Sinha. 2006. “Segmentation of pipe images for crack detection in buried sewers.” Comput.-Aided Civ. Inf. 21 (6): 395–410. https://doi.org/10.1111/j.1467-8667.2006.00445.x.
Krizhevsky, A., I. Sutskever, and G. Hinton. 2012. “Imagenet classification with deep convolutional neural networks.” In Proc., Neural Information Processing Systems Conference, 1097–1105. Salt Lake City: NeurIPS Fund.
Kumar, S., D. Abraham, M. Jahanshahi, T. Iseley, and J. Starr. 2018. “Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks.” Autom. Constr. 91 (Jul): 273–283. https://doi.org/10.1016/j.autcon.2018.03.028.
Kumar, S., M. Wang, D. Abraham, M. Jahanshahi, T. Iseley, and J. Cheng. 2020. “Deep learning-based automated detection of sewer defects in CCTV videos.” J. Comput. Civ. Eng. 34 (1): 04019047. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000866.
Lepot, M., N. Stanić, and F. Clemens. 2017. “A technology for sewer pipe inspection (part 2): Experimental assessment of a new laser profiler for sewer defect detection and quantification.” Autom. Constr. 73 (Jan): 1–11. https://doi.org/10.1016/j.autcon.2016.10.010.
Li, D., A. Cong, and S. Guo. 2019. “Sewer damage detection from imbalanced CCTV inspection data using deep convolutional neural networks with hierarchical classification.” Autom. Constr. 101 (May): 199–208. https://doi.org/10.1016/j.autcon.2019.01.017.
Ling, J., X. Yang, S. Lau, X. Wang, S. Luo, and V. Lee. 2020. “Ding L. Automated pavement crack detection and segmentation based on two-step convolutional neural network.” Comput.-Aided Civ. Inf. 35 (11): 1291–1305. https://doi.org/10.1111/mice.12622.
Long, J., S. Evan, and D. Trevor. 2015. “Fully convolutional networks for semantic segmentation.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 3431–3440. New York: IEEE.
Mashford, J., M. Rahilly, P. Davis, and S. Burn. 2010. “A morphological approach to pipe image interpretation based on segmentation by support vector machine.” Autom. Constr. 19 (7): 875–883. https://doi.org/10.1016/j.autcon.2010.06.001.
McKim, R., and S. Sinha. 1999. “Condition assessment of underground sewer pipes using a modified digital image processing paradigm.” Tunnelling Underground Space Technol. 14 (2): 29–37. https://doi.org/10.1016/S0886-7798(00)00021-3.
Mei, Q., and G. Mustafa. 2020. “A cost effective solution for pavement crack inspection using cameras and deep neural networks.” Constr. Build. Mater. 256 (Sep): 119397. https://doi.org/10.1016/j.conbuildmat.2020.119397.
Moradi, S., T. Zayed, F. Nasiri, and F. Golkhoo. 2020. “Automated anomaly detection and localization in sewer inspection videos using proportional data modeling and deep learning-based text recognition.” J. Infrastrut. Syst. 26 (3): 04020018. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000553.
Navab-Kashani, R., L. Gay, and A. Bayat. 2015. “Productivity improvement of sewer CCTV inspection through time study and route optimization.” J. Constr. Eng. Manage. 141 (6): 04015009. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000976.
Nishikawa, T., J. Yoshida, T. Sugiyama, and Y. Fujino. 2012. “Concrete crack detection by multiple sequential image filtering.” Comput.-Aided Civ. Inf. 27 (1): 29–47. https://doi.org/10.1111/j.1467-8667.2011.00716.x.
Pan, G., Y. Zheng, S. Guo, and Y. Lv. 2020. “Automatic sewer pipe defect semantic segmentation based on improved U-Net.” Autom. Constr. 119 (Nov): 103383. https://doi.org/10.1016/j.autcon.2020.103383.
Pluvinage, G. 2006. “Pipe-defect assessment based on the limit analysis, failure-assessment diagram, and subcritical crack growth.” Mater. Sci. 42 (1): 127–139. https://doi.org/10.1007/s11003-006-0065-x.
Ren, S., K. He, R. Girshick, and J. Sun. 2017. “Faster R-CNN: Towards real-time object detection with region proposal networks.” IEEE Trans. Pattern Anal. 39 (6): 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Ren, Y., J. Huang, Z. Hong, W. Lu, J. Yin, L. Zou, and X. Shen. 2019. “Image-based concrete crack detection in tunnels using deep fully convolutional networks.” J. Infrastrut. Syst. 234 (Feb): 117367. https://doi.org/10.1016/j.conbuildmat.2019.117367.
Shannon, C. 1948. “A mathematical theory of communication.” Bell Syst. Tech. J. 27 (4): 623–656. https://doi.org/10.1002/j.1538-7305.1948.tb00917.x.
Simonyan, K., and A. Zisserman. 2015. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted September 4, 2015. http://arxiv.org/abs/1409.1556.
Sinha, S., and P. Fieguth. 2006. “Segmentation of buried concrete pipe images.” Autom. Constr. 15 (1): 47–57. https://doi.org/10.1016/j.autcon.2005.02.007.
Sinha, S., P. Fieguth, and M. Polak. 2010. “Computer vision techniques for automatic structural assessment of underground pipes.” Comput.-Aided Civ. Inf. 18 (2): 95–112. https://doi.org/10.1111/1467-8667.00302.
Stanić, N., M. Lepot, M. Catieau, J. Langeveld, and F. Clemens. 2017. “A technology for sewer pipe inspection (part 1): Design, calibration, corrections and potential application of a laser profiler.” Autom. Constr. 75 (Mar): 91–107. https://doi.org/10.1016/j.autcon.2016.12.005.
Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. “Going deeper with convolutions.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1–9. New York: IEEE.
Tedeschi, A., and F. Benedetto. 2017. “A real-time automatic pavement crack and pothole recognition system for mobile Android-based devices.” Adv. Eng. Inform. 32 (Apr): 11–25. https://doi.org/10.1016/j.aei.2016.12.004.
Vinayakumar, R., K. Soman, and P. Prabaharan. 2017. “Applying deep learning approaches for network traffic prediction.” In Proc., 2017 Int. Conf. on Advances in Computing, Communications and Informatics IEEE, 2353–2358. New York: IEEE.
Wang, M., and J. Cheng. 2020. “A unified convolutional neural network integrated with conditional random field for pipe defect segmentation.” Comput.-Aided Civ. Inf. 35 (2): 162–177. https://doi.org/10.1111/mice.12481.
Wang, N., Q. Zhao, S. Li, X. Zhao, and P. Zhao. 2018. “Damage classification for masonry historic structures using convolutional neural networks based on still images.” Comput.-Aided Civ. Inf. 33 (12): 1073–1089. https://doi.org/10.1111/mice.12411.
Wang, N., X. Zhao, L. Wang, and L. Zou. 2019. “Novel system for rapid investigation and damage detection in cultural heritage conservation based on deep learning.” J. Infrastrut. Syst. 25 (3): 04019020. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000499.
Wang, N., X. Zhao, Z. Zou, P. Zhao, and F. Qi. 2020. “Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning.” Comput.-Aided Civ. Inf. 35 (3): 277–291. https://doi.org/10.1111/mice.12488.
Xu, Y., G. Zhao, M. Wang, C. Wang, and D. Zhu. 2014. “Application of dual-frequency identification sonar to detection of underwater apparent defect of hydraulic structure.” Water Resour. Hydropower Eng. 35 (7): 103–106. https://doi.org/10.13928/j.cnki.wrahe.2014.07.057.
Yin, X., Y. Chen, A. Bouferguene, H. Zaman, M. Al-Hussein, and L. Kurach. 2019. “A deep learning-based framework for an automated defect detection system for sewer pipes.” Autom. Constr. 109 (Jan): 102967. https://doi.org/10.1016/j.autcon.2019.102967.
Information & Authors
Information
Published In
Journal of Infrastructure Systems
Volume 29 • Issue 1 • March 2023
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Jan 23, 2021
Accepted: Aug 15, 2022
Published online: Dec 23, 2022
Published in print: Mar 1, 2023
Discussion open until: May 23, 2023
ASCE Technical Topics:
- Buried pipes
- Case studies
- Corrosion
- Data analysis
- Detection methods
- Deterioration
- Engineering fundamentals
- Environmental engineering
- Fouling
- Infrastructure
- Lifeline systems
- Materials characterization
- Materials engineering
- Methodology (by type)
- Pipeline systems
- Pipes
- Research methods (by type)
- Sewer pipes
- Sewers
- Steel pipes
- Waste management
- Waste treatment
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.