Automated Detection of Pavement Manhole on Asphalt Pavements with an Improved YOLOX
Publication: Journal of Infrastructure Systems
Volume 29, Issue 4
Abstract
Accurate recognition and location of pavement manholes are of great significance for pavement maintenance. This paper proposes an improved You only look once X (YOLOX) for automated detection of manholes on asphalt pavements. The proposed model improves the performance of the YOLOX model in two respects. First, the channel attention mechanism is introduced to enhance the model’s adaptive feature refinement; second, a microscale detection layer is deployed in the YOLOX model to extract more essential and distinct features. The experimental results are impressive, with the improved YOLOX achieving an F1 score and overall intersection-over-union of 98.14% and 91.61%, respectively, on 250 testing images, surpassing other state-of-the-art models such as YOLOv4, Faster R-CNN, EfficientDet, and the original YOLOX. To demonstrate robustness of the proposed model, the improved YOLOX is further applied to process manhole images taken randomly by a smartphone, which differ significantly from those acquired by a laser imaging system. It is found that the improved YOLOX can also yield similar detection efficiency in different scenes, which indicates the proposed model has a strong generalization ability. Particularly, the average frame per second (FPS) of the improved YOLOX is approximately 50.74 FPS using a modern graphic processing unit (GPU) device, implying the promising potential of the proposed model in supporting real-time automated detection of pavement manholes.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
The data that support the findings of this study are available from the corresponding author upon reasonable request, including the code of the improved YOLOX.
Acknowledgments
The study presented in this article was partially supported by the National Natural Science Foundation of China (Grant No. 51208419) and Shudao Investment Group Science and Technology Program.
Author contributions: Network conception and design: Hang Zhang, Zishuo Dong, and Jing Shang; data preparation: Anzheng He, Yang Liu, Kelvin C. P. Wang, and Zhihao Lin; experiment design and analysis of results: Hang Zhang and Allen A. Zhang; and manuscript preparation: Hang Zhang and Allen A. Zhang.
References
Bochkovskiy, A., C. Y. Wang, and H. Y. M. Liao. 2020. “YOLOv4: Optimal speed and accuracy of object detection.” Preprint, submitted April 23, 2020. https://arxiv.org/abs/2004.10934.
Chang, L.-Y. 2014. “Analysis of effects of manhole covers on motorcycle driver maneuvers: A nonparametric classification tree approach.” Traffic Inj. Prev. 15 (2): 206–212. https://doi.org/10.1080/15389588.2013.792110.
Chen, C., S. Chandra, Y. F. Han, and H. Seo. 2022a. “Deep learning-based thermal image analysis for pavement defect detection and classification considering complex pavement conditions.” Remote Sens. 14 (1): 106. https://doi.org/10.3390/rs14010106.
Chen, C., H. Seo, C. H. Jun, and Y. Zhao. 2022b. “Pavement crack detection and classification based on fusion feature of LBP and PCA with SVM.” Int. J. Pavement Eng. 23 (9): 3274–3283. https://doi.org/10.1080/10298436.2021.1888092.
Chen, C., H. Seo, and Y. Zhao. 2022c. “A novel pavement transverse cracks detection model using WT-CNN and STFT-CNN for smartphone data analysis.” Int. J. Pavement Eng. 23 (12): 4372–4384. https://doi.org/10.1080/10298436.2021.1945056.
De Boer, P.-T., D. P. Kroese, S. Mannor, and R. Y. Rubinstein. 2005. “A tutorial on the cross-entropy method.” Ann. Oper. Res. 134 (1): 19–67. https://doi.org/10.1007/s10479-005-5724-z.
Dhiman, A., and R. Klette. 2020. “Pothole detection using computer vision and learning.” IEEE Trans. Intell. Transp. Syst. 21 (8): 3536–3550. https://doi.org/10.1109/TITS.2019.2931297.
Fei, Y., K. C. P. Wang, A. Zhang, C. Chen, J. Q. Li, Y. Liu, G. Yang, and B. Li. 2019. “Pixel-level cracking detection on 3D asphalt pavement images through deep-learning-based CrackNet-V.” IEEE Trans. Intell. Transp. Syst. 21 (1): 273–284. https://doi.org/10.1109/TITS.2019.2891167.
Ge, Z., S. Liu, F. Wang, Z. Li, and J. Sun. 2021. “YOLOX: Exceeding YOLO series in 2021.” Preprint, submitted July 18, 2021. https://arxiv.org/abs/2107.08430.
He, K., X. Zhang, S. Ren, and J. Sun. 2015. “Spatial pyramid pooling in deep convolutional networks for visual recognition.” IEEE Trans. Pattern Anal. Mach. Intell. 37 (9): 1904–1916. https://doi.org/10.1109/TPAMI.2015.2389824.
Huyan, J., W. Li, S. Tighe, Z. Xu, and J. Zhai. 2020. “CrackU-net: A novel deep convolutional neural network for pixelwise pavement crack detection.” Struct. Control Health Monit. 27 (8): e2551. https://doi.org/10.1002/stc.2551.
Lin, T. Y., P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. 2017a. “Feature pyramid networks for object detection.” In Proc., 2017 IEEE Conf. on Computer Vision and Pattern Recognition, 936–944. New York: IEEE. https://doi.org/10.1109/CVPR.2017.106.
Lin, T. Y., P. Goyal, R. Girshick, K. He, and P. Dollár. 2017b. “Focal loss for dense object detection.” In Proc., 2017 IEEE Conf. on Int. Conf. on Computer Vision, 2999–3007. New York: IEEE. https://doi.org/10.1109/ICCV.2017.324.
Liu, S., L. Qi, H. Qin, J. Shi, and J. Jia. 2018. “Path aggregation network for instance segmentation.” In Proc., 2018 IEEE Conf. on Computer Vision and Pattern Recognition, 8759–8768. New York: IEEE. https://doi.org/10.1109/CVPR.2018.00913.
Qing, L., K. Yang, W. Tan, and J. Li. 2020. “Automated detection of manhole covers in MLS point clouds using a deep learning approach.” In Proc., 2020 IEEE Conf. on Int. Geoscience and Remote Sensing Symp. (IGARSS), 1580–1583. New York: IEEE. https://doi.org/10.1109/IGARSS39084.2020.9324137.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” In Proc., 29th Annual Conf. on Neural Information Processing Systems, 28. Montreal: Neural Information Processing Systems.
Rezatofighi, H., N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese. 2019. “Generalized intersection over union: A metric and a loss for bounding box regression.” In Proc., 2019 IEEE Conf. on Computer Vision and Pattern Recognition, 658–666. New York: IEEE. https://doi.org/10.1109/CVPR.2019.00075.
Samarasekera, A. C. J., and H. Shin. 2019. “Cooperative nano communication in the THz gap frequency range using wireless power transfer.” KSII Trans. Internet Inf. Syst. 13 (10): 5035–5057. https://doi.org/10.3837/tiis.2019.10.012.
Santos, A., et al. 2020. “Storm-drain and manhole detection using the RetinaNet method.” Sensors 20 (16): 4450. https://doi.org/10.3390/s20164450.
Suong, L. K., and J. Kwon. 2018. “Detection of potholes using a deep convolutional neural network.” J. Univ. Comput. Sci. 24 (9): 1244–1257.
Tan, M., R. Pang, and Q. V. Le. 2020. “EfficientDet: Scalable and efficient object detection.” Preprint, submitted November 20, 2019. https://arxiv.org/abs/1911.09070.
Tang, Y., A. A. Zhang, L. Luo, G. Wang, and E. Yang. 2021. “Pixel-level pavement crack segmentation with encoder-decoder network.” Measurement 184 (Nov): 109914. https://doi.org/10.1016/j.measurement.2021.109914.
Wada, T., and S. Kawai. 2018. “Detection of 3D reflector code on guardrail by using infrared laser radar for road information acquisition.” IEICE Trans. Fundam. Electron. Commun. Comput. Sci. 101 (9): 1320–1322. https://doi.org/10.1587/transfun.E101.A.1320.
Wang, C. Y., H. Y. M. Liao, Y. H. Wu, P. Y. Chen, J. W. Hsieh, and I. H. Yeh. 2020a. “CSPNet: A new backbone that can enhance learning capability of CNN.” In Proc., 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops, 1571–1580. New York: IEEE. https://doi.org/10.1109/CVPRW50498.2020.00203.
Wang, J., et al. 2020b. “Deep high-resolution representation learning for visual recognition.” IEEE Trans. Pattern Anal. Mach. Intell. 43 (10): 3349–3364. https://doi.org/10.1109/TPAMI.2020.2983686.
Wang, Q., B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu. 2020c. “ECA-Net: Efficient channel attention for deep convolutional neural networks.” Preprint, submitted October 8, 2019. https://arxiv.org/abs/1910.03151.
Wu, H., L. Yao, Z. Xu, Y. Li, X. Ao, Q. Chen, Z. Li, and B. Meng. 2019. “Road pothole extraction and safety evaluation by integration of point cloud and images derived from mobile mapping sensors.” Adv. Eng. Inf. 42 (Oct): 100936. https://doi.org/10.1016/j.aei.2019.100936.
Yu, Y., H. Guan, and Z. Ji. 2015. “Automated detection of urban road manhole covers using mobile laser scanning data.” IEEE Trans. Intell. Transp. Syst. 16 (6): 3258–3269. https://doi.org/10.1109/TITS.2015.2413812.
Zhang, A., K. C. Wang, Y. Fei, Y. Liu, C. Chen, G. Yang, J. Q. Li, E. Yang, and S. Qiu. 2019. “Automated pixel-level pavement crack detection on 3D asphalt surfaces with a recurrent neural network.” Comput.-Aided Civ. Infrastruct. Eng. 34 (3): 213–229. https://doi.org/10.1111/mice.12409.
Zhang, A., K. C. Wang, Y. Fei, Y. Liu, S. Tao, C. Chen, J. Q. Li, and B. Li. 2018. “Deep learning-based fully automated pavement crack detection on 3D asphalt surfaces with an improved CrackNet.” J. Comput. Civ. Eng. 32 (5): 04018041. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000775.
Zhang, A., K. C. Wang, B. Li, E. Yang, X. Dai, Y. Peng, Y. Fei, Y. Liu, J. Q. Li, and C. Chen. 2017. “Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network.” Comput.-Aided Civ. Infrastruct. Eng. 32 (10): 805–819. https://doi.org/10.1111/mice.12297.
Zhang, H., L. An, V. W. Chu, D. A. Stow, X. Liu, and Q. Ding. 2021. “Learning adjustable reduced downsampling network for small object detection in urban environments.” Remote Sens. 13 (18): 3608. https://doi.org/10.3390/rs13183608.
Zhao, Q., Z. Li, W. Hu, X. Meng, and H. Zhang. 2019. “Driving comfort evaluation for manhole covers and pavement around manholes.” Adv. Mater. Sci. Eng. 2019 (Oct): 1–10. https://doi.org/10.1155/2019/1293619.
Zheng, Z., P. Wang, W. Liu, J. Li, R. Ye, and D. Ren. 2020. “Distance-IoU loss: Faster and better learning for bounding box regression.” In Vol. 34 of Proc., 34th Conf. on Artificial Intelligence, 12993–13000. New York: Association for the Advancement of Artificial Intelligence.
Zou, Q., Z. Zhang, Q. Li, X. Qi, Q. Wang, and S. Wang. 2018. “DeepCrack: Learning hierarchical convolutional features for crack detection.” IEEE Trans. Image Process. 28 (3): 1498–1512. https://doi.org/10.1109/TIP.2018.2878966.
Information & Authors
Information
Published In
Journal of Infrastructure Systems
Volume 29 • Issue 4 • December 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Feb 24, 2023
Accepted: Jun 8, 2023
Published online: Jul 26, 2023
Published in print: Dec 1, 2023
Discussion open until: Dec 26, 2023
ASCE Technical Topics:
- Architectural engineering
- Artificial intelligence and machine learning
- Asphalt pavements
- Automation and robotics
- Building management
- Channels (waterway)
- Computer programming
- Computer vision and image processing
- Computing in civil engineering
- Engineering fundamentals
- Hydraulic engineering
- Hydraulic structures
- Infrastructure
- Maintenance and operation
- Methodology (by type)
- Neural networks
- Pavements
- Systems engineering
- Transportation engineering
- Water and water resources
- Waterways
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.