Detection of Asphalt Pavement Cracks Based on Vision Transformer Improved YOLO V5
Publication: Journal of Transportation Engineering, Part B: Pavements
Volume 149, Issue 2
Abstract
Automatic and rapid detection of pavement cracks is one of the important tasks for the highway department. This study proposed an improved model of you only look once (YOLO) V5 integrated with the vision transformer (ViT) that can calculate the attention weights of image regions and form a new feature map with weights. The ViT module was added to the neck of YOLO V5 to improve the speed and accuracy of the model. 1944 asphalt pavement images were collected for testing. The test results showed that the proposed model obtained high accuracy and speed for longitudinal, transverse, and fatigue cracks and was capable of real-time detection. The ViT-improved YOLO V5m obtained 0.872 in mAP(0.5), and the detection time for a single image is 11.9 ms. The study also investigated the pavement crack detection performance of the models in a rainfall environment. All models did not have satisfying detection capabilities in rainfall conditions, but the developed YOLO V5 had better performance.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statements
All data that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work was sponsored by the projects found by the National Natural Science Foundation of China (NSFC) under Grant Nos. 51978163 and 52208439 and the Jiangsu Nature Science Foundation under Grant BK20200468.
References
Aziz, L., M. S. B. Salam, U. U. Sheikh, and S. Ayub. 2020. “Exploring deep learning-based architecture, strategies, applications and current trends in generic object detection: A comprehensive review.” IEEE Access 8 (Sep): 170461–170495. https://doi.org/10.1109/ACCESS.2020.3021508.
Behzadian, A., et al. 2022. “The 1st data science for pavements challenge.” Preprint, submitted June 10, 2022. https://arxiv.org/abs/2206.04874.
Bello, I., B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le. 2019. “Attention augmented convolutional networks.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 3286–3295. Los Alamitos, CA: IEEE Computer Society.
Bochkovskiy, A., C.-Y. Wang, and H.-Y. Liao. 2020. “Yolov4: Optimal speed and accuracy of object detection.” Preprint, submitted April 23, 2020. https://arxiv.org/abs/2004.10934.
Dosovitskiy, A., et al. 2020. “An image is worth 16×16 words: Transformers for image recognition at scale.” Preprint, submitted October 22, 2020. https://arxiv.org/abs/2010.11929.
Du, Y. C., N. Pan, Z. H. Xu, F. W. Deng, Y. Shen, and H. Kang. 2021. “Pavement distress detection and classification based on YOLO network.” Int. J. Pavement Eng. 22 (13): 1659–1672. https://doi.org/10.1080/10298436.2020.1714047.
Feng, X. R., L. Y. Xiao, W. Li, L. L. Pei, Z. Y. Sun, Z. D. Ma, H. Shen, and H. Y. Ju. 2020. “Pavement crack detection and segmentation method based on improved deep learning fusion model.” Math. Probl. Eng. 2020 (Dec): 1–22. https://doi.org/10.1155/2020/8515213.
Girshick, R. 2015. “Fast R-CNN.” In Proc., IEEE Int. Conf. on Computer Vision, 1440–1448. Los Alamitos, CA: IEEE Computer Society.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2014. “Rich feature hierarchies for accurate object detection and semantic segmentation.” In Proc., 27th IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 580–587. Los Alamitos, CA: IEEE Computer Society.
He, K. M., X. Y. Zhang, S. Q. Ren, and J. Sun. 2014. “Spatial pyramid pooling in deep convolutional networks for visual recognition.” In Proc., 13th European Conf. on Computer Vision (ECCV), 346–361. Cham, Switzerland: Springer.
Hu, G. X., B. L. Hu, Z. Yang, L. Huang, and P. Li. 2021. “Pavement crack detection method based on deep learning models.” Wireless Commun. Mobile Comput. 2021 (May): 1–3. https://doi.org/10.1155/2021/5573590.
Huyan, J., W. Li, S. Tighe, J. Z. Zhai, Z. C. Xu, and Y. Chen. 2019. “Detection of sealed and unsealed cracks with complex backgrounds using I (A) deep convolutional neural network.” Autom. Constr. 107 (Nov): 102946. https://doi.org/10.1016/j.autcon.2019.102946.
Ibragimov, E., H. J. Lee, J. J. Lee, and N. Kim. 2022. “Automated pavement distress detection using region based convolutional neural networks.” Int. J. Pavement Eng. 23 (6): 1981–1992. https://doi.org/10.1080/10298436.2020.1833204.
Jeong, D. J. 2020. “Road damage detection using YOLO with smartphone images.” In Proc., 8th IEEE Int. Conf. on Big Data (Big Data), 5559–5562. Los Alamitos, CA: IEEE Computer Society.
Jeong, J. H., J. S. Lim, Y. C. Suh, and J. H. Nam. 2014. “Development of performance criteria for Korean pavement warranty specification.” J. Perform. Constr. Facil. 28 (2): 402–411. https://doi.org/10.1061/(ASCE)CF.1943-5509.0000405.
Kalfarisi, R., Z. Y. Wu, and K. Soh. 2020. “Crack detection and segmentation using deep learning with 3D reality mesh model for quantitative assessment and integrated visualization.” J. Comput. Civ. Eng. 34 (3): 04020010. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000890.
Kang, L. W., C. W. Lin, and Y. H. Fu. 2012. “Automatic single-image-based rain streaks removal via image decomposition.” IEEE Trans. Image Process. 21 (4): 1742–1755. https://doi.org/10.1109/TIP.2011.2179057.
Li, S. W., X. Y. Gu, X. R. Xu, D. W. Xu, T. J. Zhang, Z. Liu, and Q. Dong. 2021. “Detection of concealed cracks from ground penetrating radar images based on deep learning algorithm.” Constr. Build. Mater. 273 (Mar): 121949. https://doi.org/10.1016/j.conbuildmat.2020.121949.
Liu, C. Q., J. Li, J. Gao, Z. Q. Gao, and Z. J. Chen. 2021. “Combination of pixel-wise and region-based deep learning for pavement inspection and segmentation.” Int. J. Pavement Eng. 23 (9): 3011–3023. https://doi.org/10.1080/10298436.2021.1877704.
Liu, S., L. Qi, H. F. Qin, J. P. Shi, and J. Y. Jia. 2018. “Path aggregation network for instance segmentation.” In Proc., 31st IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 8759–8768. Los Alamitos, CA: IEEE Computer Society.
Liu, W., D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. 2016. “Ssd: Single shot multibox detector.” In Proc., European Conf. on Computer Vision, 21–37. Cham, Switzerland: Springer.
Ma, D., H. Y. Fang, B. H. Xue, F. M. Wang, M. A. Msekh, and C. L. Chan. 2020. “Intelligent detection model based on a fully convolutional neural network for pavement cracks.” CMES-Comput. Model. Eng. Sci. 123 (3): 1267–1291. https://doi.org/10.32604/cmes.2020.09122.
Mandal, V., L. Uong, and Y. Adu-Gyamfi. 2018. “Automated road crack detection using deep convolutional neural networks.” In Proc., IEEE Int. Conf. on Big Data (Big Data), 5212–5215. Los Alamitos, CA: IEEE Computer Society.
Nie, M., and K. Wang. 2018. “Pavement distress detection based on transfer learning.” In Proc., 5th Int. Conf. on Systems and Informatics (ICSAI), 435–439. Piscataway, NJ: IEEE.
Ren, S. Q., K. M. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” In Proc., 29th Annual Conf. on Neural Information Processing Systems (NIPS). La Jolla, CA: Neural Information Processing Systems.
Ronneberger, O., P. Fischer, and T. Brox. 2015. “U-net: Convolutional networks for biomedical image segmentation.” In Proc., Int. Conf. on Medical Image Computing and Computer-Assisted Intervention, 234–241. Cham, Switzerland: Springer.
Song, L., and X. C. Wang. 2021. “Faster region convolutional neural network for automated pavement distress detection.” Road Mater. Pavement Des. 22 (1): 23–41. https://doi.org/10.1080/14680629.2019.1614969.
Tran, T. S., V. P. Tran, H. J. Lee, J. M. Flores, and V. P. Le. 2020. “A two-step sequential automated crack detection and severity classification process for asphalt pavements.” Int. J. Pavement Eng. 23 (6): 2019–2033. https://doi.org/10.1155/2021/5573590.
Ultralytics. 2022. “YOLOV5.” Accessed February 13, 2022. https://github.com/ultralytics/yolov5.
Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. “Attention is all you need.” In Proc., Advances in Neural Information Processing Systems, 5998–6008. La Jolla, CA: Neural Information Processing Systems.
Wang, C.-Y., H.-Y. M. Liao, Y.-H. Wu, P.-Y. Chen, J.-W. Hsieh, and I.-H. Yeh. 2020. “CSPNet: A new backbone that can enhance learning capability of CNN.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops, 390–391. Los Alamitos, CA: IEEE Computer Society.
Wang, T., K. Gopalakrishnan, O. Smadi, and A. K. Somani. 2018a. “Automated shape-based pavement crack detection approach.” Transport 33 (3): 598–608. https://doi.org/10.3846/transport.2018.1559.
Wang, W. Z., B. Wu, S. X. Yang, and Z. X. Wang. 2018b. “Road damage detection and classification with faster R-CNN.” In Proc., IEEE Int. Conf. on Big Data (Big Data), 5220–5223. Los Alamitos, CA: IEEE Computer Society.
Yang, X., J. C. Guan, L. Ding, Z. P. You, V. C. S. Lee, M. R. M. Hasan, and X. Y. Cheng. 2021. “Research and applications of artificial neural network in pavement engineering: A state-of-the-art review.” J. Traffic Transp. Eng. 8 (6): 1000–1021. https://doi.org/10.1016/j.jtte.2021.03.005.
Zhang, H., and V. M. Patel. 2018. “Density-aware single image de-raining using a multi-stream dense network.” In Proc., 31st IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 695–704. Los Alamitos, CA: IEEE Computer Society.
Zhao, H., J. Jia, and V. Koltun. 2020. “Exploring self-attention for image recognition.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 10076–10085. Los Alamitos, CA: IEEE Computer Society.
Information & Authors
Information
Published In
Journal of Transportation Engineering, Part B: Pavements
Volume 149 • Issue 2 • June 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Feb 23, 2022
Accepted: Dec 26, 2022
Published online: Feb 21, 2023
Published in print: Jun 1, 2023
Discussion open until: Jul 21, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Yuxuan Li, Shangyu Sun, Weidong Song, Jinhe Zhang, Qiaoshuang Teng, CrackYOLO: Rural Pavement Distress Detection Model with Complex Scenarios, Electronics, 10.3390/electronics13020312, 13, 2, (312), (2024).
- Yazhen Sun, Haixiang Tang, Huaizhi Zhang, Automatic Detection of Pavement Marking Defects in Road Inspection Images Using Deep Learning, Journal of Performance of Constructed Facilities, 10.1061/JPCFEV.CFENG-4619, 38, 2, (2024).
- Haitao Li, Tao Peng, Ningguo Qiao, Zhiwei Guan, Xinyun Feng, Peng Guo, Tingting Duan, Jinfeng Gong, CrackTinyNet: A novel deep learning model specifically designed for superior performance in tiny road surface crack detection, IET Intelligent Transport Systems, 10.1049/itr2.12497, (2024).
- Saúl Cano-Ortiz, Lara Lloret Iglesias, Pablo Martinez Ruiz del Árbol, Pedro Lastra-González, Daniel Castro-Fresno, An end-to-end computer vision system based on deep learning for pavement distress detection and quantification, Construction and Building Materials, 10.1016/j.conbuildmat.2024.135036, 416, (135036), (2024).
- Binghan Xue, Jianglin Gao, Songtao Hu, Yan Li, Jianguo Chen, Rui Pang, Ground Penetrating Radar Image Recognition for Earth Dam Disease Based on You Only Look Once v5s Algorithm, Water, 10.3390/w15193506, 15, 19, (3506), (2023).
- Iason Katsamenis, Athanasios Sakelliou, Nikolaos Bakalos, Eftychios Protopapadakis, Christos Klaridopoulos, Nikolaos Frangakis, Matthaios Bimpas, Dimitris Kalogeras, Deep transformer networks for precise pothole segmentation tasks, Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, 10.1145/3594806.3596560, (596-602), (2023).
- Deren Zhu, Linfeng Dai, Peirong Du, CCE-YOLOv5s: An Improved YOLOv5 Model for UAV Small Target Detection, 2023 IEEE 5th International Conference on Civil Aviation Safety and Information Technology (ICCASIT), 10.1109/ICCASIT58768.2023.10351744, (824-829), (2023).