Automatic Detection of Pavement Marking Defects in Road Inspection Images Using Deep Learning
Publication: Journal of Performance of Constructed Facilities
Volume 38, Issue 2
Abstract
Pavement markings serve to convey information to drivers, regulate driving behavior, and effectively mitigate traffic congestion and reduce accidents. Nonetheless, due to traffic exposure and temperature stress, pavement markings may develop defects to diverse degrees. Consequently, the inspection and maintenance of pavement markings has been paid high attention. Traditional manual detection methods prove time-consuming, subjective, and present security risks. Therefore, we employed four object detection models [You Only Look Once version 5 (YOLOv5), YOLOv7, faster region convolutional neural networks (Faster R-CNN) with visual geometry group laboratory (VGG), and Faster R-CNN with residual network (ResNet)] to achieve intelligent recognition of pavement marking defects through deep learning. Each model underwent 1,000 epochs of training and utilized 2,000 annotated road inspection images. Through data augmentation, module optimization, and anchor redesign, these models can locate pavement markings and classify their defects. The accuracy and efficiency of the model were evaluated by mean average precision (mAP) and frames per second. In addition, we introduced evaluation indicators that focused on defect types to assist in selecting models with high applicability in detecting markings. Among these models, the optimized Faster R-CNN with VGG as the backbone network has an mAP of 93.96% and can detect over 28 images per second, which meets the engineering requirements.
Practical Applications
Pavement markings play a crucial role in guiding driver behavior, and they significantly contribute to alleviating traffic congestion and reducing traffic accidents. As the demand for autonomous driving technology increases, the maintenance of pavement markings has become more important. To address this, it is essential to determine which pavement markings require maintenance through inspection. However, traditional manual inspection methods suffer from issues such as low efficiency, high cost, and safety hazards. In this work, we developed four models utilizing artificial intelligence to automatically detect pavement markings in road inspection vehicle images. This approach significantly reduces the need for manual operation and ensures efficient and safe detection of pavement markings. The automatic detection results accurately match the actual position of the pavement markings, thus providing valuable guidance for maintenance work.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this research are available from the corresponding author upon reasonable request.
Acknowledgments
The authors greatly acknowledge the financial and data support from the Key R&D Program of Zhejiang Province (No. 2022C03180), Liaoning Provincial Educational Department (LJKZZ20220080), and Shandong Provincial Institute of Transportation Sciences (Grant No. SDJKY2020-014).
References
Adu-Gyamfi, Y., T. Tienaah, N. Attoh-Okine, and C. Kambhamettu. 2014. “Functional evaluation of pavement condition using a complete vision system.” J. Transp. Eng. 140 (9): 04014040. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000638.
Bahar, G., M. Masliah, T. Erwin, E. Tan, and E. Hauer. 2006. Pavement marking materials and markers: Real-world relationship between retroreflectivity and safety over time. Thousand Oaks, CA: SAGE.
Bochkovskiy, A., C.-Y. Wang, and H.-Y. M. Liao. 2020. “Yolov4: Optimal speed and accuracy of object detection.” Preprint, submitted April 23, 2020. http://arxiv.org/abs/2004.10934.
Carlson, P. J., E. S. Park, and C. K. Andersen. 2009. “Benefits of pavement markings.” Transp. Res. Rec. 2107: 59–68. https://doi.org/10.3141/2107-06.
Carlson, P. J., G. Schertz, C. Satterfield, K. W. Falk, and T. Taylor. 2014. Methods for maintaining pavement marking retroreflectivity. Washington, DC: Federal Highway Administration, Office of Safety.
Chen, T., Z. Chen, Q. Shi, and X. Huang. 2015. “Road marking detection and classification using machine learning algorithms.” In Proc., 2015 IEEE Intelligent Vehicles Symp. (IV), 617–621. New York: IEEE.
Chimba, D., E. Kidando, and M. Onyango. 2018. “Evaluating the service life of thermoplastic pavement markings: Stochastic approach.” J. Transp. Eng. Part B. Pavements 144 (3): 04018029. https://doi.org/10.1061/JPEODX.0000055.
Hani, A., L. R. Andrea, and S. Robert. 2021. “Detecting of pavement marking defects using faster R-CNN.” J. Perform. Constr. Facil. 35 (4): 04021035. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001606.
Hawkins, N. 2021. “Developing pavement marking management systems: A theoretical model framework based on the experiences of the US transportation agencies.” Infrastructures 6 (2): 8. https://doi.org/10.3390/infrastructures6020018.
Hu, J.-W., B.-Y. Zheng, C. Wang, C.-H Zhao, X.-L. Hou, Q. Pan, and Z. Xu. 2020. “A survey on multi-sensor fusion based obstacle detection for intelligent ground vehicles in off-road environments.” Front. Inf. Technol. Electron. Eng. 21 (5): 675–692. https://doi.org/10.1631/FITEE.1900518.
Karwa, V., and E. T. Donnell. 2011. “Predicting pavement marking retroreflectivity using artificial neural networks: Exploratory analysis.” J. Transp. Eng. 137 (2): 91–103. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000194.
Kawano, M., K. Mikami, S. Yokoyama, T. Yonezawa, and J. Nakazawa. 2017. “Road marking blur detection with drive recorder.” In Proc., 2017 IEEE Int. Conf. on Big Data (Big Data), 4092–4097. New York: IEEE.
Li, X., J. Li, X. Hu, and J. Yang. 2019. “Line-CNN: End-to-end traffic line detection with line proposal unit.” IEEE Trans. Intell. Transp. Syst. 21 (1): 248–258. https://doi.org/10.1109/TITS.2019.2890870.
Luo, H., J. Li, L. Cai, and M. Wu. 2023. “STrans-YOLOX: Fusing swin transformer and YOLOX for automatic pavement crack detection.” Appl. Sci. 13 (3): 1999. https://doi.org/10.3390/app13031999.
Markow, M. J. 2008. “Current asset management practices applied to pavement markings.” Transp. Res. Rec. 2055 (1): 78–86. https://doi.org/10.3141/2055-10.
Pan, X., J. Shi, P. Luo, X. Wang, and X. Tang. 2018. “Spatial as deep: Spatial CNN for traffic scene understanding.” In Proc., AAAI Conf. on Artificial Intelligence. Palo Alto, CA: Association for the Advancement of Artificial Intelligence.
Parajuli, A., M. Celenk, and H. B. Riley. 2013. “Robust lane detection in shadows and low illumination conditions using local gradient features.” Open J. Appl. Sci. 3 (1): 68. https://doi.org/10.4236/ojapps.2013.31B014.
Pike, A. M., and B. Bommanayakanahalli. 2018. “Development of a pavement marking life cycle cost tool.” Transp. Res. Rec. 2672 (12): 148–157. https://doi.org/10.1177/0361198118758012.
Qiao, W., Q. Liu, X. Wu, B. Ma, and G. Li. 2021. “Automatic pixel-level pavement crack recognition using a deep feature aggregation segmentation network with a scSE attention mechanism module.” Sensors 21 (9): 2902. https://doi.org/10.3390/s21092902.
Qin, Z., H. Wang, and X. Li. 2016. “Ultra fast structure-aware deep lane detection.” In Proc., Computer Vision–ECCV 2020: 16th European Conf., Part XXIV 16, 276–291. Cham, Switzerland: Springer.
Rasdorf, W., J. E. Hummer, E. A. Harris, and W. E. Sitzabee. 2009. “IT issues for the management of high-quantity, low-cost assets.” J. Comput. Civ. Eng. 23 (2): 91–99. https://doi.org/10.1061/(ASCE)0887-3801(2009)23:2(91).
Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 779–788. Washington, DC: IEEE Computer Society.
Redmon, J., and A. Farhadi. 2017. “YOLO9000: Better, faster, stronger.” Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 7263–7271. Washington, DC: IEEE Computer Society.
Redmon, J., and A. Farhadi. 2018. “Yolov3: An incremental improvement.” Preprint, submitted April 3, 2018. http://arxiv.org/abs/1804.02767.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster r-CNN: Towards real-time object detection with region proposal networks.” IEEE Trans. Pattern Anal. Mach. Intell. 39 (6): 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Sasidharan, L., V. Karwa, and E. T. Donnell. 2009. “Use of pavement marking degradation models to develop a pavement marking management system.” Public Works Manage. Policy 14 (2): 148–173. https://doi.org/10.1177/1087724X09349513.
Sitzabee, W. E., J. E. Hummer, and W. Rasdorf. 2009a. “Pavement marking degradation modeling and analysis.” J. Infrastruct. Syst. 15 (3): 190–199. https://doi.org/10.1061/(ASCE)1076-0342(2009)15:3(190).
Sitzabee, W. E., W. Rasdorf, J. E. Hummer, and H. A. Devine. 2009b. “Data integration of pavement markings: A case in transportation asset management.” J. Comput. Civ. Eng. 23 (5): 288–298. https://doi.org/10.1061/(ASCE)0887-3801(2009)23:5(288).
Wan, H., L. Gao, M. Su, Q. Sun, and L. Huang. 2021. “Attention-based convolutional neural network for pavement crack detection.” Adv. Mater. Sci. Eng. 2021 (1): 1–13. https://doi.org/10.1155/2021/5520515.
Wang, C.-Y., A. Bochkovskiy, and H.-Y. M. Liao. 2023a. “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 7464–7475. Washington, DC: IEEE Computer Society.
Wang, S., X. Chen, and Q. Dong. 2023b. “Detection of asphalt pavement cracks based on vision transformer improved YOLO V5.” J. Transp. Eng. Part B. Pavements 149 (2): 04023004. https://doi.org/10.1061/JPEODX.PVENG-1180.
Information & Authors
Information
Published In
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Jul 17, 2023
Accepted: Nov 8, 2023
Published online: Jan 9, 2024
Published in print: Apr 1, 2024
Discussion open until: Jun 9, 2024
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.