Automated Anomaly Detection and Localization in Sewer Inspection Videos Using Proportional Data Modeling and Deep Learning–Based Text Recognition
Publication: Journal of Infrastructure Systems
Volume 26, Issue 3
Abstract
In recent decades, closed circuit television (CCTV) has been the most used tool for visually inspecting the internal condition of pipelines. However, CCTV inspection requires long videos to be observed and analyzed by certified inspectors, which is time-consuming, labor-intensive, and error-prone. This paper proposes a novel approach for automated anomaly detection and localization in sewer CCTV inspection videos. The developed algorithms employ three-dimensional (3D) Scale Invariant Feature Transform (SIFT) to extract spatio-temporal features in sewer CCTV videos. Anomaly detection is performed using a one-class support vector machine (OC-SVM) trained by frames without defects to model states considered normal and to classify outliers to this model as anomalous frames. Then, the identified anomalous frames are located by recognizing included text information in them using an end-to-end text recognition approach. The proposed localization approach is divided into two main steps: text detection using maximally stable extremal regions (MSER) algorithm and text recognition using a deep convolutional neural network (CNN). Extracting and localizing the suspicious frames out of these videos for further analysis can reduce the time and cost of detection because thousands of normal frames would be detached in the inspection process. The proposed model performance showed acceptable viability, because the testing accuracy was 92.3% in anomaly detection and 86.6% for frame localization in sewer inspection video frames.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code generated or used during the research are available from the corresponding author by request (computational models and codes).
Acknowledgments
This research was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC), grant number N1009.
References
ASCE. 2017. Report card for America’s infrastructure. Reston, VA: ASCE.
Chen, H., S. S. Tsai, G. Schroth, D. M. Chen, R. Grzeszczuk, and B. Girod. 2011. “Robust text detection in natural images with edge-enhanced maximally stable extremal regions.” In Proc., 2011 18th IEEE Int. Conf. on Image Processing, 2609–2612. New York: IEEE.
Cheng, J. C. P., and M. Wang. 2018. “Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques.” Autom. Constr. 95 (Nov): 155–171. https://doi.org/10.1016/j.autcon.2018.08.006.
Cortes, C., and V. Vapnik. 1995. “Support-vector networks.” Mach. Learning 20 (3): 273–297. https://doi.org/10.1007/BF00994018.
Epshtein, B., E. Ofek, and Y. Wexler. 2010. “Detecting text in natural scenes with stroke width transform.” In Proc., 2010 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 2963–2970. New York: IEEE.
Erfani, S. M., S. Rajasegarar, S. Karunasekera, and C. Leckie. 2016. “High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning.” Pattern Recognit. 58 (Oct): 121–134. https://doi.org/10.1016/j.patcog.2016.03.028.
FCM (Federation of Canadian Municipalities). 2019. Canadian infrastructure report card (CIRC). Ottawa: FCM.
Gao, W., X. Zhang, L. Yang, and H. Liu. 2010. “An improved sobel edge detection.” In Proc., 2010 3rd Int. Conf. on Computer Science and Information Technology, 67–71. New York: IEEE.
González, Á, L. M. Bergasa, J. J. Yebes, and S. Bronte. 2012. “Text location in complex images.” In Proc., 21st Int. Conf. on Pattern Recognition (ICPR2012), 617–620. New York: IEEE.
Graves, A., M. Liwicki, B. Horst, J. Schmidhuber, and F. Santiago. 2008. “Unconstrained on-line handwriting recognition with recurrent neural networks.” In Advances in neural information processing systems, edited by J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis. Red Hook, NY: Curran Associates.
Guo, W., L. Soibelman, and J. Garrett. 2009a. “Automated defect detection for sewer pipeline inspection and condition assessment.” Autom. Constr. 18 (5): 587–596. https://doi.org/10.1016/j.autcon.2008.12.003.
Guo, W., L. Soibelman, and J. Garrett. 2009b. “Automated defect detection in urban wastewater pipes using invariant features found in video images.” In Proc., Construction Research Congress, Reston, VA: ASCE.
Halfawy, M., and J. Hengmeechai. 2014. “Automated defect detection in sewer closed circuit television images using histograms of oriented gradients and support vector machine.” Autom. Constr. 38 (Mar): 1–13. https://doi.org/10.1016/j.autcon.2013.10.012.
Halfawy, M., and J. Hengmeechai. 2015. “Integrated vision-based system for automated defect detection in sewer closed circuit television inspection videos.” J. Comput. Civ. Eng. 29 (1): 04014024. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000312.
Hassan, S. I., L. M. Dang, I. Mehmood, S. Im, C. Choi, J. Kang, Y. Park, and H. Moon. 2019. “Underground sewer pipe condition assessment based on convolutional neural networks.” Autom. Constr. 106 (Oct): 102849. https://doi.org/10.1016/j.autcon.2019.102849.
Joachims, T. 1998. Making large-scale SVM learning practical. LS VIII-report. Dortmund, Germany: Univ. of Dortmund.
Kecman, V. 2001. Learning and soft computing: Support vector machines, neural networks, and fuzzy logic models. Cambridge, MA: MIT Press.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2012. “ImageNet classification with deep convolutional neural networks.” In Advances in neural information processing systems 25, edited by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, 1097–1105. Red Hook, NY: Curran Associates.
Kumar, S. S., D. M. Abraham, M. R. Jahanshahi, T. Iseley, and J. Starr. 2018. “Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks.” Autom. Constr. 91 (Jul): 273–283. https://doi.org/10.1016/j.autcon.2018.03.028.
Kumar, S. S., W. Mingzhu, D. M. Abraham, M. R. Jahanshahi, I. Tom, and J. C. Cheng. 2020. “Deep learning–based automated detection of sewer defects in CCTV videos” J. Comput. Civ. Eng. 34 (1): 04019047. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000866.
Lee, J., P. Lee, S. Lee, A. Yuille, and C. Koch. 2011. “AdaBoost for text detection in natural scene.” In Proc., Int. Conf. on Document Analysis and Recognition, 429–434. New York: IEEE.
Li, Y., and H. Lu. 2012. “Scene text detection via stroke width.” In Proc., 21st Int. Conf. on Pattern Recognition (ICPR2012), 681–684. New York: IEEE.
Long, S., X. He, and C. Ya. 2018. “Scene text detection and recognition: The deep learning era.” Preprint, submitted November 10, 2018. http://arXiv.org/abs/1811.04256.
Lowe, D. G. 1999. “Object recognition from local scale-invariant features.” In Proc., Int. Conf. on Computer Vision-Volume 2—Volume 2, 1150. Washington, DC: IEEE Computer Society.
Lowe, D. G. 2004. “Distinctive Image features from scale-invariant keypoints.” Int. J. Comput. Vision 60 (2): 91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94.
Lucas, S. 2005. “ICDAR 2005 text locating competition results.” In Vol. 1 of Proc., 8th Int. Conf. on Document Analysis and Recognition (ICDAR’05), 80–84. New York: IEEE.
Mashford, J., M. Rahilly, P. Davis, and S. Burn. 2010. “A morphological approach to pipe image interpretation based on segmentation by support vector machine.” Autom. Constr. 19 (7): 875–883. https://doi.org/10.1016/j.autcon.2010.06.001.
Matas, J., O. Chum, M. Urban, and T. Pajdla. 2004. “Robust wide-baseline stereo from maximally stable extremal regions.” Image Vision Comput. 22 (10): 761–767. https://doi.org/10.1016/j.imavis.2004.02.006.
Meijer, D., L. Scholten, F. Clemens, and A. Knobbe. 2019. “A defect classification methodology for sewer image sets with convolutional neural networks.” Autom. Constr. 104 (Aug): 281–298. https://doi.org/10.1016/j.autcon.2019.04.013.
Mikolajczyk, K., T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir, and L. V. Gool. 2005. “A comparison of affine region detectors.” Int. J. Comput. Vision 65 (1): 43–72. https://doi.org/10.1007/s11263-005-3848-x.
Moradi, S., and T. Zayed. 2017. “Real-time defect detection in sewer closed circuit television inspection videos.” In Proc., ASCE Pipelines 2017, 295–307. Reston, VA: ASCE.
Moradi, S., T. Zayed, and F. Golkhoo. 2018. “Automated sewer pipeline inspection using computer vision techniques.” In Proc., Pipelines 2018: Condition Assessment, Construction, and Rehabilitation, 582–587. Reston, VA: ASCE.
Moradi, S., T. Zayed, and F. Golkhoo. 2019. “Review on computer aided sewer pipeline defect detection and condition assessment.” Infrastructures 4 (1): 10. https://doi.org/10.3390/infrastructures4010010.
Moselhi, O., and T. Shehab-Eldeen. 1999. “Automated detection of surface defects in water and sewer pipes.” Autom. Constr. 8 (5): 581–588. https://doi.org/10.1016/S0926-5805(99)00007-2.
Myrans, J., R. Everson, and Z. Kapelan. 2018. “Automated detection of faults in sewers using CCTV image sequences.” Autom. Constr. 95 (Nov): 64–71. https://doi.org/10.1016/j.autcon.2018.08.005.
Ni, D., Y. P. Chui, Y. Qu, X. Yang, J. Qin, T. Wong, S. S. H. Ho, and P. A. Heng. 2009. “Reconstruction of volumetric ultrasound panorama based on improved 3D SIFT.” Computerized Med. Imaging Graphics 33 (7): 559–566. https://doi.org/10.1016/j.compmedimag.2009.05.006.
Oliva, A., and A. Torralba. 2001. “Modeling the shape of the scene: A holistic representation of the spatial envelope.” Int. J. Comput. Vision 42 (3): 145–175. https://doi.org/10.1023/A:1011139631724.
Olson, D. L., and D. Delen. 2008. Advanced data mining techniques. Berlin: Springer.
Opitz, M., M. Diem, S. Fiel, F. Kleber, and R. Sablatnig. 2014. “End-to-end text recognition using local ternary patterns, MSER and deep convolutional nets.” In Proc., 11th IAPR International Workshop on Document Analysis Systems, 186–190. New York: IEEE.
Pan, Y., X. Hou, and C. Liu. 2009. “Text localization in natural scene images based on conditional random field.” In Proc., 10th Int. Conf. on Document Analysis and Recognition, 6–10. New York: IEEE.
Sarshar, N., M. Halfawy, and J. Hengmeechai. 2009. “Video processing techniques for assisted CCTV inspection and condition rating of sewers.” J. Water Manage. Model. https://doi.org/10.14796/JWMM.R235-08.
Schölkopf, B., J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson. 2001. “Estimating the support of a high-dimensional distribution.” Neural Comput. 13 (7): 1443–1471. https://doi.org/10.1162/089976601750264965.
Scovanner, P., S. Ali, and M. Shah. 2007. “A 3-dimensional sift descriptor and its application to action recognition.” In Proc., 15th ACM Int. Conf. on Multimedia, 357–360. New York: ACM.
Shahid, N., I. H. Naqvi, and S. B. Qaisar. 2015. “One-class support vector machines: analysis of outlier detection for wireless sensor networks in harsh environments.” Artif. Intell. Rev. 43 (4): 515–563. https://doi.org/10.1007/s10462-013-9395-x.
Sinha, S. K., and P. W. Fieguth. 2006. “Segmentation of buried concrete pipe images.” Autom. Constr. 15 (1): 47–57. https://doi.org/10.1016/j.autcon.2005.02.007.
Steinwart, I., and A. Christmann. 2008. Support vector machines. New York: Springer.
Su, T. C. 2015. “Segmentation of crack and open joint in sewer pipelines based on CCTV inspection images.” In Proc., Int. Conf. on Circuits and Systems (CAS 2015). Red Hook, NY: Curran Associates.
Su, T. C., and M. D. Yang. 2015. “Application of morphological segmentation to leaking defect detection in sewer pipelines.” In Proc., 2015 AASRI Int. Conf. on Circuits and Systems, 8686–8704. Paris: Atlantis Press.
Su, T. C., M. D. Yang, T. Wu, and J. Lin. 2011. “Morphological segmentation based on edge detection for sewer pipe defects on CCTV images.” Expert Syst. Appl. 38 (10): 13094–13114. https://doi.org/10.1016/j.eswa.2011.04.116.
Wang, M., and J. C. P. Cheng. 2019. “A unified convolutional neural network integrated with conditional random field for pipe defect segmentation.” Comput. -Aided Civ. Infrastruct. Eng. 35 (2): 162–177. https://doi.org/10.1111/mice.12481.
Xie, Q., D. Li, J. Xu, Z. Yu, and J. Wang. 2019. “Automatic detection and classification of sewer defects via hierarchical deep learning.” IEEE Trans. Autom. Sci. Eng. 16 (4): 1836–1847. https://doi.org/10.1109/TASE.2019.2900170.
Yin, F., Y. Wu, X. Zhang, and C. Liu. 2017. “Scene text recognition with sliding convolutional character models.” Preprint, submitted September 6, 2017. http://arXiv.org/abs/1709.01727.
Information & Authors
Information
Published In
Journal of Infrastructure Systems
Volume 26 • Issue 3 • September 2020
Copyright
©2020 American Society of Civil Engineers.
History
Received: Aug 7, 2019
Accepted: Jan 23, 2020
Published online: Apr 24, 2020
Published in print: Sep 1, 2020
Discussion open until: Sep 24, 2020
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.