Multilabel Sewer Pipe Defect Recognition with Mask Attention Feature Enhancement and Label Correlation Learning
Publication: Journal of Computing in Civil Engineering
Volume 39, Issue 1
Abstract
The coexistence of multiple defect categories as well as the substantial class imbalance problem significantly impair the detection of sewer pipeline defects. To solve this problem, a multilabel pipe defect recognition method is proposed based on mask attention-guided feature enhancement and label correlation learning. The proposed method can achieve current approximate state-of-the-art classification performance using just 1/16 of the Sewer-ML training data set and exceeds the current best method by 11.87% in terms of F2 metric on the full data set, while also proving the superiority of the model. The major contribution of this study is the development of a more efficient model for identifying and locating multiple defects in sewer pipe images for a more accurate sewer pipeline condition assessment. Moreover, by employing class activation maps, our method can accurately pinpoint multiple defect categories in the image, demonstrating strong model interpretability.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work was supported in part by NSF of China under Grant No. 61903164 and in part by NSF of Jiangsu Province in China under Grants BK20191427.
References
Alejo, D., F. Caballero, and L. Merino. 2017. “Rgbd-based robot localization in sewer networks.” In Proc., 2017 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 4070–4076. New York: IEEE.
Bahnsen, C. H., A. S. Johansen, M. P. Philipsen, J. W. Henriksen, K. Nasrollahi, and T. B. Moeslund. 2021. “3D sensors for sewer inspection: A quantitative review and analysis.” Sens. 21 (7): 2553. https://doi.org/10.3390/s21072553.
Bryant, L. F. 2022. “Water and wastewater infrastructure.” In Women in infrastructure, 417–440. Cham, Switzerland: Springer.
Chen, K., H. Hu, C. Chen, L. Chen, and C. He. 2018. “An intelligent sewer defect detection method based on convolutional neural network.” In Proc., 2018 IEEE Int. Conf. on Information and Automation (ICIA), 1301–1306. New York: IEEE.
Chen, K., H. Li, C. Li, X. Zhao, S. Wu, Y. Duan, and J. Wang. 2022. “An automatic defect detection system for petrochemical pipeline based on cycle-gan and yolo v5.” Sensors 22 (20): 7907. https://doi.org/10.3390/s22207907.
Chen, T., M. Xu, X. Hui, H. Wu, and L. Lin. 2019a. “Learning semantic-specific graph representation for multi-label image recognition.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 522–531. New York: IEEE.
Chen, Z.-M., X.-S. Wei, P. Wang, and Y. Guo. 2019b. “Multi-label image recognition with graph convolutional networks.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 5177–5186. New York: IEEE.
Dang, L. M., H. Wang, Y. Li, T. N. Nguyen, and H. Moon. 2022. “DefectTR: End-to-end defect detection for sewage networks using a transformer.” Constr. Build. Mater. 325 (Mar): 126584. https://doi.org/10.1016/j.conbuildmat.2022.126584.
Demidov, D., M. H. Sharif, A. Abdurahimov, H. Cholakkal, and F. S. Khan. 2023. “Salient mask-guided vision transformer for fine-grained classification.” Preprint, submitted May 11, 2023. http://arxiv.org/abs/2305.07102.
Haurum, J. B., M. M. Allahham, M. S. Lynge, K. S. Henriksen, I. A. Nikolov, and T. B. Moeslund. 2021. “Sewer defect classification using synthetic point clouds.” In Proc., VISIGRAPP (5: VISAPP), 891–900. Setúbal, Portugal: Science and Technology Publications.
Haurum, J. B., M. Madadi, S. Escalera, and T. B. Moeslund. 2022a. “Multi-scale hybrid vision transformer and sinkhorn tokenizer for sewer defect classification.” Autom. Constr. 144 (Dec): 104614. https://doi.org/10.1016/j.autcon.2022.104614.
Haurum, J. B., M. Madadi, S. Escalera, and T. B. Moeslund. 2022b. “Multi-task classification of sewer pipe defects and properties using a cross-task graph neural network decoder.” In Proc., IEEE/CVF Winter Conf. on Applications of Computer Vision, 2806–2817. New York: IEEE.
Haurum, J. B., and T. B. Moeslund. 2020. “A survey on image-based automation of CCTV and SSET sewer inspections.” Autom. Constr. 111 (Mar): 103061. https://doi.org/10.1016/j.autcon.2019.103061.
Haurum, J. B., and T. B. Moeslund. 2021. “Sewer-ML: A multi-label sewer defect classification dataset and benchmark.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 13456–13467. New York: IEEE.
He, K., G. Gkioxari, P. Dollár, and R. Girshick. 2018. “Mask R-CNN.” In Proc., IEEE Int. Conf. on Computer Vision, 2961–2969. New York: IEEE.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Hu, C., B. Dong, H. Shao, J. Zhang, and Y. Wang. 2023. “Toward purifying defect feature for multilabel sewer defect classification.” IEEE Trans. Instrum. Meas. 72 (Feb): 1–11.
Kingma, D. P., and J. Ba. 2014. “Adam: A method for stochastic optimization.” Preprint, submitted April 24, 2015. http://arxiv.org/abs/1412.6980.
Kirillov, A., et al. 2023. “Segment anything.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 4015–4026. New York: IEEE.
Lanchantin, J., T. Wang, V. Ordonez, and Y. Qi. 2021. “General multi-label image classification with transformers.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 16478–16488. New York: IEEE.
Li, Y., H. Wang, L. M. Dang, H.-K. Song, and H. Moon. 2022. “Vision-based defect inspection and condition assessment for sewer pipes: A comprehensive survey.” Sensors 22 (7): 2722. https://doi.org/10.3390/s22072722.
Lin, T.-Y., P. Goyal, R. Girshick, K. He, and P. Dollár. 2017. “Focal loss for dense object detection.” In Proc., IEEE Int. Conf. on Computer Vision, 2980–2988. New York: IEEE.
Liu, S., L. Zhang, X. Yang, H. Su, and J. Zhu. 2021a. “Query2label: A simple transformer way to multi-label classification.” Preprint, submitted July 22, 2021. http://arxiv.org/abs/2107.10834.
Liu, Z., Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. 2021b. “Swin transformer: Hierarchical vision transformer using shifted windows.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 10012–10022. New York: IEEE.
Nie, L., T. Chen, Z. Wang, W. Kang, and L. Lin. 2022. “Multi-label image recognition with attentive transformer-localizer module.” Multimedia Tools Appl. 81 (6): 7917–7940. https://doi.org/10.1007/s11042-021-11818-8.
og Spildevandsforening. 2005. Fotomanualen: Beregning af fysisk indeks ved tv-inspektion. Skanderborg, Denmark: Dansk Vand og Spildevandsforening.
og Spildevandsforening. 2010. Fotomanualen: Tv-inspektion af afløbsledninger, 6. Skanderborg, Denmark: Dansk Vand og Spildevandsforening.
Pu, T., M. Sun, H. Wu, T. Chen, L. Tian, and L. Lin. 2023. “Semantic representation and dependency learning for multi-label image recognition.” Neurocomputing 526 (Mar): 121–130. https://doi.org/10.1016/j.neucom.2023.01.018.
Ridnik, T., E. Ben-Baruch, N. Zamir, A. Noy, I. Friedman, M. Protter, and L. Zelnik-Manor. 2021a. “Asymmetric loss for multi-label classification.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 82–91. New York: IEEE.
Ridnik, T., H. Lawen, A. Noy, E. Ben Baruch, G. Sharir, and I. Friedman. 2021b. “TResNet: High performance GPU-dedicated architecture.” In Proc., IEEE/CVF Winter Conf. on Applications of Computer Vision, 1400–1409. New York: IEEE.
Ryali, C., et al. 2023. “Hiera: A hierarchical vision transformer without the bells-and-whistles.” Preprint, submitted June 1, 2023. http://arxiv.org/abs/2306.00989.
Tao, M., L. Wan, H. Wang, and T. Su. 2022. “CAFEN: A correlation-aware feature enhancement network for sewer defect identification.” In Proc., 2022 21st Int. Symp. on Communications and Information Technologies (ISCIT), 204–209. New York: IEEE.
Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. “Attention is all you need.” In Advances in neural information processing systems, 30. San Diego: NeurIPS.
Wei, Y., W. Xia, M. Lin, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan. 2015. “HCP: A flexible CNN framework for multi-label image classification.” IEEE Trans. Pattern Anal. Mach. Intell. 38 (9): 1901–1907. https://doi.org/10.1109/TPAMI.2015.2491929.
Wu, H., B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang. 2021. “CvT: Introducing convolutions to vision transformers.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 22–31. New York: IEEE.
Wu, H., C. Xu, and H. Liu. 2022. “S-MAT: Semantic-driven masked attention transformer for multi-label aerial image classification.” Sensors 22 (14): 5433. https://doi.org/10.3390/s22145433.
Xie, Q., D. Li, J. Xu, Z. Yu, and J. Wang. 2019. “Automatic detection and classification of sewer defects via hierarchical deep learning.” IEEE Trans. Autom. Sci. Eng. 16 (4): 1836–1847. https://doi.org/10.1109/TASE.2019.2900170.
Xu, S., Y. Li, J. Hsiao, C. Ho, and Z. Qi. 2022. “A dual modality approach for (zero-shot) multi-label classification.” Preprint, submitted August 19, 2022. http://arxiv.org/abs/2208.09562.
Yuan, J., Y. Zhang, Z. Shi, X. Geng, J. Fan, and Y. Rui. 2023. “Balanced masking strategy for multi-label image classification.” Neurocomputing 522 (Feb): 64–72. https://doi.org/10.1016/j.neucom.2022.11.022.
Zhang, W., C. Liu, L. Zeng, B. Ooi, S. Tang, and Y. Zhuang. 2023. “Learning in imperfect environment: Multi-label classification with long-tailed distribution and partial labels.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 1423–1432. New York: IEEE.
Zhou, W., P. Dou, T. Su, H. Hu, and Z. Zheng. 2023. “Feature learning network with transformer for multi-label image classification.” Pattern Recognit. 136 (Apr): 109203. https://doi.org/10.1016/j.patcog.2022.109203.
Zhu, X., J. Cao, J. Ge, W. Liu, and B. Liu. 2022. “Two-stream transformer for multi-label image classification.” In Proc., 30th ACM Int. Conf. on Multimedia, 3598–3607. New York: IEEE.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 39 • Issue 1 • January 2025
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Jan 24, 2024
Accepted: Jul 16, 2024
Published online: Oct 18, 2024
Published in print: Jan 1, 2025
Discussion open until: Mar 18, 2025
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.