Multilabel CNN Model for Asphalt Distress Classification
Publication: Journal of Computing in Civil Engineering
Volume 38, Issue 1
Abstract
One of the most challenging tasks in pavement management and rehabilitation is to detect and classify different distress types from images collected during field surveys. In this paper, a multilabel convolutional neural network (CNN) model for classifying asphalt distress is proposed. Unlike typical CNN models that classify a single object per image, the proposed model can detect and classify multiple distress types per image, without prior knowledge of the distress location. The model can classify the distress types into four categories: alligator cracking, block cracking, longitudinal/transverse cracking, and pothole. The proposed model was trained and tested on a real data set comprising 42,520 images using different pretrained architectures with various hyperparameter combinations. The results demonstrate the robustness of the proposed model and its potential for crack detection and localization using weakly supervised machine learning methods that can cope with partially labeled data sets.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
The data that support the findings of this study are available from Geokom Ltd., but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are available from the authors upon reasonable request and with the permission of Geokom Ltd.
Acknowledgments
The images data set was kindly provided by Geokom Ltd., which performs the field surveys for the Israel National Company for Transport Infrastructure.
Author contributions: All authors contributed to the study conception and design. Mai Sirhan performed material preparation, data collection, analysis, and writing of the first draft. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
References
Bang, S., S. Park, H. Kim, and H. Kim. 2019. “Encoder-decoder network for pixel-level road 326 crack detection in black-box images.” Comput.-Aided Civ. Infrastruct. Eng. 34 (8): 713–727. https://doi.org/10.1111/mice.12440.
Bochkovskiy, A., C. Y. Wang, and H. Y. Liao. 2020. “Yolov4: Optimal speed and accuracy of object detection.” Preprint, submitted April 23, 2018. http://arxiv.org/abs/2004.10934.
Cao, M. T., Q. V. Tran, N. M. Nguyen, and K. Chang. 2020a. “Survey on performance of deep learning models for detecting road damages using multiple dashcam image resources.” Adv. Eng. Inf. 46 (Mar): 101182. https://doi.org/10.1016/j.aei.2020.101182.
Cao, W., Q. Liu, and Z. He. 2020b. “Review of pavement defect detection methods.” IEEE Access 334 (8): 575. https://doi.org/10.1109/ACCESS.2020.2966881.
Dong, Z., J. Wang, B. Cui, D. Wang, and X. Wang. 2020. “Patch-based weakly supervised semantic segmentation network for crack detection.” Constr. Build. Mater. 258 (Oct): 120291. https://doi.org/10.1016/j.conbuildmat.2020.120291.
Du, Y., N. Pan, Z. Xu, F. Deng, Y. Shen, and H. Kang. 2021. “Pavement distress detection and classification based on YOLO network.” Int. J. Pavement Eng. 22 (13): 1659–1672. https://doi.org/10.1080/10298436.2020.1714047.
Ge, Z., S. Liu, F. Wang, Z. Li, and J. Sun. 2021. “YOLOX: Exceeding YOLO Series in 2021.” Preprint, submitted July 18, 2021. http://arxiv.org/abs/210708430.
Girshick, R. 2015. “Fast r-cnn.” In Proc., IEEE Int. Conf. on computer vision, 1440–1448. New York: IEEE.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2015. “Region-based convolutional networks for accurate object detection and segmentation.” IEEE Trans. Pattern Anal. Mach. Intell. 38 (1): 142–158. https://doi.org/10.1109/TPAMI.2015.2437384.
Gopalakrishnan, K. 2018. “Deep learning in data-driven pavement image analysis and automated distress detection: A review.” Data 3 (3): 28. https://doi.org/10.3390/data3030028.
Gopalakrishnan, K., S. K. Khaitan, A. Choudhary, and A. Agrawal. 2017. “Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection.” Constr. Build. Mater. 157 (Dec): 322–330. https://doi.org/10.1016/j.conbuildmat.2017.09.110.
He, K., G. Gkioxari, P. Dollár, and R. Girshick. 2017. “Mask R-CNN.” In Proc., IEEE. New York: IEEE.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Hou, Y., Q. Li, Q. Han, B. Peng, L. Wang, X. Gu, and D. Wang. 2021. “MobileCrack: Object classification in asphalt pavements using an adaptive lightweight deep learning.” J. Transp. Eng. 147 (1): 92. https://doi.org/10.1061/JPEODX.0000245.
Hsieh, Y. A., and Y. J. Tsai. 2020. “Machine learning for crack detection: Review and model performance comparison.” J. Comput. Civ. Eng. 34 (5): 38. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000918.
Li, B., K. C. Wang, A. Zhang, E. Yang, and G. Wang. 2020. “Automatic classification of pavement crack using deep convolutional neural network.” Int. J. Pavement Eng. 21 (4): 457–463. https://doi.org/10.1080/10298436.2018.1485917.
Liu, W., D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, and A. Berg. 2016. “SSD: Single shot multibox detector.” In Proc., European Conf. on Computer Vision (ECCV), 21–37. New York: Springer.
Liu, Y., Y. Yuan, and J. Liu. 2021. “Deep learning model for imbalanced multi-label surface defect classification.” Meas. Sci. Technol. 33 (3): 601. https://doi.org/10.1088/1361-6501/ac41a6.
Maeda, H., Y. Sekimoto, T. Seto, T. Kashiyama, and H. Omata. 2018. “Road damage detection and classification using deep neural networks with smartphone images.” Comput.-Aided Civ. Infrastruct. Eng. 33 (12): 1127–1141. https://doi.org/10.1111/mice.12387.
Park, S., S. Bang, H. Kim, and H. Kim. 2019. “Patch-based crack detection in black box images using convolutional neural networks.” J. Comput. Civ. Eng. 33 (3): 17. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000831.
Peraka, N. S., K. P. Biligiri, and S. Kalidindi. 2021. “Development of a multi-distress detection system for asphalt pavements: Transfer learning-based approach.” Transp. Res. Rec. 2675 (10): 538–553. https://doi.org/10.1177/03611981211012001.
Protopapadakis, E., I. Katsamenis, and A. Doulamis. 2020. “Multi-label deep learning models for continuous monitoring of road infrastructures.” In Proc., 13th ACM Int. Conf. on Pervasive Technologies Related to Assistive Environments, 1–7. New York: Association for Computing Machinery.
Redmon, J., and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” In Proc., IEEE Conf. on computer Vision and Pattern Recognition, 779–788. New York: IEEE.
Redmon, J., and A. Farhadi. 2017. “YOLO9000: Better, faster, stronger.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Redmon, J., and A. Farhadi. 2018. “YOLOv3: An incremental improvement.” Preprint, submitted April 8, 2018. https://doi.org/10.48550/arXiv.1804.02767.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” In Advances in neural information processing systems, 28. Red Hook, NY: Curran Associates.
Sirhan, M., S. Bekhor, and A. Sidess. 2022. “Implementation of deep neural networks for pavement condition index prediction.” J. Transp. Eng. 148 (1): 04021070-1–04021070-11. https://doi.org/10.1061/JPEODX.0000333.
Tran, T. S., V. P. Tran, H. J. Lee, J. M. Flores, and V. Le. 2022. “A two-step sequential automated crack detection and severity classification process for asphalt pavements.” Int. J. Pavement Eng. 23 (6): 2019–2033. https://doi.org/10.1080/10298436.2020.1836561.
Yamashita, R., M. Nishio, R. K. Do, and K. Togashi. 2018. “Convolutional neural networks: An overview and application in radiology.” Insights Imaging 9 (Aug): 611–629. https://doi.org/10.1007/s13244-018-0639-9.
Yusof, N. A., M. K. Osman, M. H. Noor, I. A. Tahir, and N. M. Yusof. 2018. “Crack detection and classification in asphalt pavement images using deep convolution neural network.” In Proc., 8th IEEE Int. Conf. on Control System, Computing and Engineering (ICCSCE), 227–232. New York: IEEE.
Yusof, N. A. M., A. Ibrahim, M. H. M. Noor, N. M. Tahir, N. M. Yusof, N. Z. Abidin, and M. K. Osman. 2019. “Deep convolution neural network for crack detection on asphalt pavement.” J. Phys.: Conf. Ser. 1349 (1): 20. https://doi.org/10.1088/1742-6596/1349/1/012020.
Zhang, A., K. C. Wang, B. Li, Y. E. Dai, X. Peng, Y. Fei, Y. Liu, Y. Li, J. Q. Chen, and C. Chen. 2017. “Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network.” Comput.-Aided Civ. Infrastruct. Eng. 32 (10): 805–819. https://doi.org/10.1111/mice.12297.
Zhang, L., F. Yang, Y. D. Zhang, and Y. Zhu. 2016. “Road crack detection using deep convolutional neural network.” In Proc., 2016 IEEE Int. Conf. on Image Processing (ICIP), 3708–3712. New York: IEEE.
Zou, Q., Y. Cao, Q. Li, Q. Mao, and S. Wang. 2012. “CrackTree: Automatic crack detection from pavement images.” Pattern Recognit. Lett. 33 (3): 227–238. https://doi.org/10.1016/j.patrec.2011.11.004.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 38 • Issue 1 • January 2024
Copyright
© 2023 American Society of Civil Engineers.
History
Received: May 10, 2023
Accepted: Aug 30, 2023
Published online: Oct 20, 2023
Published in print: Jan 1, 2024
Discussion open until: Mar 20, 2024
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.