Convolutional Neural Networks–Based Model for Automated Sewer Defects Detection and Classification
Publication: Journal of Water Resources Planning and Management
Volume 147, Issue 7
Abstract
Automated detection and classification of sewer defects can complement the conventional labor-intensive sewer inspection process by providing an essential tool to classify sewer defects in a more efficient, accurate, and consistent way. This paper presents a convolutional neural networks (CNNs)–based model to automatically detect and classify six most commonly observed sewer defects (i.e., cracks, disjoints, obstacles, residential walls, tree roots, and normal categories) obtained from multisource CCTV images under various circumstances. Data augmentation techniques (including geometric and color transformations) are applied to enhance the model performance. The proposed CNN model is further compared with a state-of-the-art solution (retraining the SqueezeNet using defect images) by adopting transfer learning technique. An average prediction accuracy of 90% is achieved, indicating that the investigated defects can be well recognized by the model without any expert knowledge of sewer detection. There is a higher degree of confidence in predicting tree roots and disjoints, followed by residential walls and cracks. Results show that the prediction accuracy has increased by 15% thanks to data augmentation. Despite the transferred SqueezeNet model achieved a higher accuracy (95%), it cost almost 13 times the computation time of the CNN model. The study demonstrates the feasibility of the deep learning technology in the automated classification of sewer defects and advances the knowledge in the research field.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
The training and testing dataset that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This research was funded by the National Natural Science Foundation of China (Grant No. 51809049), the Science and Technology Program of Guangzhou, China (Grant No. 201804010406).
References
Cha, Y.-J., W. Choi, and O. Büyüköztürk. 2017. “Deep learning-based crack damage detection using convolutional neural networks.” Comput.-Aided Civ. Infrastruct. Eng. 32 (5): 361–378. https://doi.org/10.1111/mice.12263.
Chen, Y., S. Zhong, K. Chen, S. Chen, and S. Zheng. 2019. “Automated detection of sewer pipe defects based on cost-sensitive convolutional neural network.” In Proc., 2019 2nd Int. Conf. on Signal Processing and Machine Learning, 8–17. Hangzhou, China: Association for Computing Machinery.
Cheng, J. C. P., and M. Wang. 2018. “Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques.” Autom. Constr. 95: 155–171. https://doi.org/10.1016/j.autcon.2018.08.006.
Dirksen, J., F. H. L. R. Clemens, H. Korving, F. Cherqui, P. Le Gauffre, T. Ertl, H. Plihal, K. Müller, and C. T. M. Snaterse. 2013. “The consistency of visual sewer inspection data.” Struct. Infrastruct. Eng. 9 (3): 214–228. https://doi.org/10.1080/15732479.2010.541265.
Douarre, C., C. F. Crispim-Junior, A. Gelibert, L. Tougne, and D. Rousseau. 2019. “Novel data augmentation strategies to boost supervised segmentation of plant disease.” Comput. Electron. Agric. 165: 104967. https://doi.org/10.1016/j.compag.2019.104967.
Gao, Y., and K. M. Mosalam. 2018. “Deep transfer learning for image-based structural damage recognition.” Comput.-Aided Civ. Infrastruct. Eng. 33 (9): 748–768. https://doi.org/10.1111/mice.12363.
Goodfellow, I., Y. Bengio, and A. Courville. 2016. Deep learning,. New York: MIT Press.
Gu, J., Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, and J. Cai. 2018. “Recent advances in convolutional neural networks.” Pattern Recognit. 77 (May): 354–377. https://doi.org/10.1016/j.patcog.2017.10.013.
Gueziri, H. E., M. J. McGuffin, and C. Laporte. 2016. “A generalized graph reduction framework for interactive segmentation of large images.” Comput. Vision Image Understanding 150 (Sep): 44–57. https://doi.org/10.1016/j.cviu.2016.05.009.
Halfawy, M. R., and J. Hengmeechai. 2014. “Efficient algorithm for crack detection in sewer images from closed-circuit television inspections.” J. Infrastruct. Syst. 20 (2): 04013014. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000161.
Halfawy Mahmoud, R., and J. Hengmeechai. 2014. “Efficient algorithm for crack detection in sewer images from closed-circuit television inspections.” J. Infrastruct. Syst. 20 (2): 04013014. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000161.
Hassan, S. I., L. M. Dang, I. Mehmood, S. Im, C. Choi, J. Kang, Y.-S. Park, and H. Moon. 2019. “Underground sewer pipe condition assessment based on convolutional neural networks.” Autom. Constr. 106 (Oct): 102849. https://doi.org/10.1016/j.autcon.2019.102849.
Hawari, A., M. Alamin, F. Alkadour, M. Elmasry, and T. Zayed. 2018. “Automated defect detection tool for closed circuit television (cctv) inspected sewer pipelines.” Autom. Constr. 89 (May): 99–109. https://doi.org/10.1016/j.autcon.2018.01.004.
Iandola, F. N., S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer. 2016. “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and model size.” Preprint, submitted November 4, 2016. http://arxiv.org/abs/1602.07360.
Inoue, H. 2018. Data augmentation by pairing samples for images classification. Ithaca, NY: Cornell University.
Iyer, S., and S. K. Sinha. 2006. “Segmentation of pipe images for crack detection in buried sewers.” Comput.-Aided Civ. Infrastruct. Eng. 21 (6): 395–410. https://doi.org/10.1111/j.1467-8667.2006.00445.x.
Jiang, J. C., J. Z. Liu, C. X. Cheng, J. Z. Huang, and A. K. Xue. 2019. “Automatic estimation of urban waterlogging depths from video images based on ubiquitous reference objects.” Remote Sensing 11 (5): 587. https://doi.org/10.3390/rs11050587.
Khodabandehlou, H., G. Pekcan, and M. S. Fadali. 2019. “Vibration-based structural condition assessment using convolution neural networks.” Struct. Control Health Monit. 26 (2): e2308. https://doi.org/10.1002/stc.2308.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2012. “ImageNet classification with deep convolutional neural networks.” In Vol. 1 of Proc., 25th Int. Conf. on Neural Information Processing Systems, 1097–1105. Red Hook, NY: Curran Associates.
Kumar, S. S., and D. M. Abraham. 2019. “A deep learning based automated structural defect detection system for sewer pipelines.” In Proc., ASCE Int. Conf. on Computing in Civil Engineering 2019, 226–233. Reston, VA: ASCE.
Kumar, S. S., D. M. Abraham, M. R. Jahanshahi, T. Iseley, and J. Starr. 2018. “Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks.” Autom. Constr. 91: 273–283. https://doi.org/10.1016/j.autcon.2018.03.028.
Lecun, Y., Y. Bengio, and G. Hinton. 2015. “Deep learning.” Nature 521 (7553): 436. https://doi.org/10.1038/nature14539.
Lin, M. 2014. “Health inspection and analysis of sewer system in an area of Fuzhou City.” [In Chinese.] China Water Wasterwater 30 (9): 96–98. https://doi.org/10.19853/j.zgjsps.1000-4602.2014.09.024.
MathWorks. 2020. “Pretrained deep neural networks.” Accessed June 22, 2020. https://www.mathworks.com/help/deeplearning/ug/pretrained-convolutional-neural-networks.html?searchHighlight=Pretrained%20deep%20neural%20networks&s_tid=srchtitle.
Meijer, D., L. Scholten, F. Clemens, and A. Knobbe. 2019. “A defect classification methodology for sewer image sets with convolutional neural networks.” Autom. Constr. 104 (Aug): 281–298. https://doi.org/10.1016/j.autcon.2019.04.013.
Miguez, M. G., O. M. Rezende, and A. P. Veról. 2015. “City growth and urban drainage alternatives: Sustainability challenge.” J. Urban Plann. Dev. 141 (3): 04014026. https://doi.org/10.1061/(ASCE)UP.1943-5444.0000219.
Moradi, S., T. Zayed, and F. Golkhoo. 2018. “Automated sewer pipeline inspection using computer vision techniques.” In Proc., Pipelines 2018: Condition Assessment, Construction, and Rehabilitation, 582–587. Reston, VA: ASCE.
Myrans, J., Z. Kapelan, and R. Everson. 2016. “Automated detection of faults in wastewater pipes from CCTV footage by using random forests.” Procedia Eng. 154 (Jan): 36–41. https://doi.org/10.1016/j.proeng.2016.07.416.
Nair, V., and G. E. Hinton. 2010. “Rectified linear units improve restricted Boltzmann machines vinod nair.” In Proc., Int. Conf. on Machine Learning, 807–814. Madison, WI: Omnipress.
Rawat, W., and Z. Wang. 2017. “Deep convolutional neural networks for image classification: A comprehensive review.” Neural Comput. 29 (9): 2352–2449. https://doi.org/10.1162/neco_a_00990.
Ren, S., K. He, R. Girshick, and J. Sun. 2017. “Faster R-CNN: Towards real-time object detection with region proposal networks.” IEEE Trans. Pattern Anal. Mach. Intell. 39 (6): 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Shin, H.-C., H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura, and R. M. Summers. 2016. “Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning.” IEEE Trans. Med. Imaging 35 (5): 1285–1298. https://doi.org/10.1109/TMI.2016.2528162.
Soukup, D., and R. Huber-Mörk. 2014. “Convolutional neural networks for steel surface defect detection from photometric stereo images.” In Proc., 2014 Int. Symp. on Visual Computing, 668–677. Cham, Switzerland: Springer.
Stanić, N., J. G. Langeveld, and F. H. L. R. Clemens. 2014. “HAZard and operability (HAZOP) analysis for identification of information requirements for sewer asset management.” Struct. Infrastruct. Eng. 10 (11): 1345–1356. https://doi.org/10.1080/15732479.2013.807845.
Su, T.-C., and M.-D. Yang. 2014. “Application of morphological segmentation to leaking defect detection in sewer pipelines.” Sensors 14 (5): 8686–8704. https://doi.org/10.3390/s140508686.
Su, T.-C., M.-D. Yang, T.-C. Wu, and J.-Y. Lin. 2011. “Morphological segmentation based on edge detection for sewer pipe defects on CCTV images.” Expert Syst. Appl. 38 (10): 13094–13114. https://doi.org/10.1016/j.eswa.2011.04.116.
Tafuri, A. N., and A. Selvakumar. 2002. “Wastewater collection system infrastructure research needs in the USA.” Urban Water 4 (1): 21–29. https://doi.org/10.1016/S1462-0758(01)00070-X.
Varkarakis, V., S. Bazrafkan, and P. Corcoran. 2020. “Deep neural network and data augmentation methodology for off-axis iris segmentation in wearable headsets.” Neural Networks 121 (Jan): 101–121. https://doi.org/10.1016/j.neunet.2019.07.020.
Wang, M., and J. C. P. Cheng. 2018. “Development and improvement of deep learning based automated defect detection for sewer pipe inspection using faster R-CNN.” In Advanced computing strategies for engineering, edited by I. F. C. Smith and B. Domer, 171–192. Cham, Lausanne: Springer.
Wang, M., B. Liu, and H. Foroosh. 2018. “Look-up table unit activation function for deep convolutional neural networks.” In Proc., IEEE Winter Conf. on Applications of Computer Vision. New York: IEEE.
Wirahadikusumah, R., D. Abraham, and T. Iseley. 2001. “Challenging issues in modeling deterioration of combined sewers.” J. Infrastruct. Syst. 7 (2): 77–84. https://doi.org/10.1061/(ASCE)1076-0342(2001)7:2(77).
Xiao, Q., J. Wang, H. Chen, S. Ye, and L. Xiang. 2019. “The detection and evaluation by CCTU and rehabilitation analysis of sewer pipeline in an area of Shenzhen City.” [In Chinese] Water Wasterwater Eng. 55 (9): 109–114. https://doi.org/10.13789/j.cnki.wwe1964.2019.09.022.
Xie, Q., D. Li, J. Xu, Z. Yu, and J. Wang. 2019. “Automatic detection and classification of sewer defects via hierarchical deep learning.” [In Chinese.] IEEE Trans. Autom. Sci. Eng. 16 (4): 1836–1847. https://doi.org/10.1109/TASE.2019.2900170.
Yang, M.-D., and T.-C. Su. 2008. “Automated diagnosis of sewer pipe defects based on machine learning approaches.” Expert Syst. Appl. 35 (3): 1327–1337. https://doi.org/10.1016/j.eswa.2007.08.013.
Yin, X. F., Y. Chen, A. Bouferguene, H. Zaman, M. Al-Hussein, and L. Kurach. 2020. “A deep learning-based framework for an automated defect detection system for sewer pipes.” Autom. Constr. 109 (Jan): 102967. https://doi.org/10.1016/j.autcon.2019.102967.
Zhang, C., C.-C. Chang, and M. Jamshidi. 2020. “Concrete bridge surface damage detection using a single-stage detector.” Comput.-Aided Civ. Infrastruct. Eng. 35 (4): 389–409. https://doi.org/10.1111/mice.12500.
Zhang, X. F., Z. Y. Wang, D. Liu, and Q. Ling. 2019. “DADA: Deep adversarial data augmentation for extremely low data regime classification.” In Proc., Int. Conf. on Acoustics, Speech and Signal Processing, 2807–2811. New York: IEEE.
Information & Authors
Information
Published In
Journal of Water Resources Planning and Management
Volume 147 • Issue 7 • July 2021
Copyright
© 2021 American Society of Civil Engineers.
History
Received: May 4, 2020
Accepted: Jan 21, 2021
Published online: Apr 29, 2021
Published in print: Jul 1, 2021
Discussion open until: Sep 29, 2021
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Mansheng Lin, Shuai Teng, Gongfa Chen, David Bassir, Transfer Learning with Attributes for Improving the Landslide Spatial Prediction Performance in Sample-Scarce Area Based on Variational Autoencoder Generative Adversarial Network, Land, 10.3390/land12030525, 12, 3, (525), (2023).
- Comfort Salihu, Saeed Reza Mohandes, Ahmed Farouk Kineber, M. Reza Hosseini, Faris Elghaish, Tarek Zayed, A Deterioration Model for Sewer Pipes Using CCTV and Artificial Intelligence, Buildings, 10.3390/buildings13040952, 13, 4, (952), (2023).
- Wei Liu, Zhiyin Xie, Zhaoyang Song, Predicting Water Pipe Failures Using Deep Learning Algorithms, Journal of Infrastructure Systems, 10.1061/JITSE4.ISENG-2247, 29, 3, (2023).
- Mohammad Amini, Khalid Kaddoura, Prediction of Sewer Pipelines Using Machine Learning Techniques: A Case Study on the City of Hamilton Sewer Network, Pipelines 2023, 10.1061/9780784485033.015, (128-138), (2023).
- Shuai Teng, Xuedi Chen, Gongfa Chen, Li Cheng, Structural damage detection based on transfer learning strategy using digital twins of bridges, Mechanical Systems and Signal Processing, 10.1016/j.ymssp.2023.110160, 191, (110160), (2023).
- Zuxiang Situ, Shuai Teng, Xiaoting Liao, Gongfa Chen, Qianqian Zhou, Real-time sewer defect detection based on YOLO network, transfer learning, and channel pruning algorithm, Journal of Civil Structural Health Monitoring, 10.1007/s13349-023-00681-w, (2023).
- Mansheng Lin, Shuai Teng, Gongfa Chen, Bo Hu, Application of convolutional neural networks based on Bayesian optimization to landslide susceptibility mapping of transmission tower foundation, Bulletin of Engineering Geology and the Environment, 10.1007/s10064-023-03069-8, 82, 2, (2023).
- Qiaojuan Fu, Xiaoying Zhang, Fukang Zhao, Ruoxin Ruan, Lihua Qian, Chunnan Li, Deep Feature Extraction for Cymbidium Species Classification Using Global–Local CNN, Horticulturae, 10.3390/horticulturae8060470, 8, 6, (470), (2022).
- Qianqian Zhou, Zuxiang Situ, Shuai Teng, Weifeng Chen, Gongfa Chen, Jiongheng Su, Comparison of classic object-detection techniques for automated sewer defect detection, Journal of Hydroinformatics, 10.2166/hydro.2022.132, 24, 2, (406-419), (2022).
- Jianbing Lv, Weijun Wu, Xiaoyu Kang, Juan Huang, Gongfa Chen, Shuai Teng, Hejie Gao, Automatic Classification System of Drainage Hole Blockage Based on Convolution Neural Network Transfer Learning, Advances in Civil Engineering, 10.1155/2022/4928018, 2022, (1-16), (2022).
- See more