A Multitask Fusion Network for Region-Level and Pixel-Level Pavement Distress Detection
Publication: Journal of Transportation Engineering, Part B: Pavements
Volume 150, Issue 1
Abstract
With the development of state-of-the-art algorithms, pavement distress can already be detected automatically. However, most pavement distress detection is currently implemented as a single task, either at the region level or at the pixel level. To comprehensively assess the pavement condition, a multitask fusion model, Pavement Distress Detection Network (PDDNet), was proposed for integrated pavement distress detection at both the region level and pixel level. PDDNet was trained and tested on distress images captured via unmanned aerial vehicle (UAV), and seven types of pavement distresses were investigated and analyzed. Compared with Mask Region-based Convolutional Neural Network (R-CNN), U-Net, and W-segnet, PDDNet shows higher performance in classification, localization, and segmentation of pavement distresses. Results demonstrate that PDDNet achieves region-level and pixel-level detection of seven types of distresses with the mean average precision of 0.810 and 0.795, respectively. As a portable and lightweight device, the UAV can collect full-width pavement distress images, which helps improve the efficiency of pavement distress detection.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or codes that support the findings of this study are available from the corresponding author upon reasonable request.
References
Arya, D., H. Maeda, S. Ghosh, D. Toshniwal, A. Mraz, T. Kashiyama, and Y. Sekimoto. 2021. “Deep learning-based road damage detection and classification for multiple countries.” Autom. Constr. 132 (Dec): 103935. https://doi.org/10.1016/j.autcon.2021.103935.
Bang, S., S. Park, H. Kim, and H. Kim. 2019. “Encoder-decoder network for pixel-level road crack detection in black-box images.” Comput.-Aided Civ. Infrastruct. Eng. 34 (8): 713–727. https://doi.org/10.1111/mice.12440.
Cao, W., Q. Liu, and Z. He. 2020. “Review of pavement defect detection methods.” IEEE Access 8 (Mar): 14531–14544. https://doi.org/10.1109/ACCESS.2020.2966881.
Dahouda, M., and I. Joe. 2021. “A deep-learned embedding technique for categorical features encoding.” IEEE Access 9 (Mar): 114381–114391. https://doi.org/10.1109/ACCESS.2021.3104357.
Du, Y., N. Pan, Z. Xu, F. Deng, Y. Shen, and H. Kang. 2021. “Pavement distress detection and classification based on YOLO network.” Int. J. Pavement Eng. 22 (13): 1659–1672. https://doi.org/10.1080/10298436.2020.1714047.
Fan, R., U. Ozgunalp, B. Hosking, M. Liu, and I. Pitas. 2020. “Pothole detection based on disparity transformation and road surface modeling.” IEEE Trans. Image Process. 29 (Aug): 897–908. https://doi.org/10.1109/TIP.2019.2933750.
Han, C., T. Ma, J. Huyan, X. Huang, and Y. Zhang. 2022. “CrackW-Net: A novel pavement crack image segmentation convolutional neural network.” IEEE Trans. Intell. Transp. Syst. 23 (11): 22135–22144. https://doi.org/10.1109/TITS.2021.3095507.
He, K., G. Gkioxari, P. Dollar, and R. Girshick. 2017. “Mask R-CNN.” In Proc., IEEE Int. Conf. on Computer Vision, 2980–2988. New York: IEEE. https://doi.org/10.1109/ICCV.2017.322.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., 2016 IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE. https://doi.org/10.1109/CVPR.2016.90.
Hsieh, Y.-A., and Y. J. Tsai. 2020. “Machine learning for crack detection: Review and model performance comparison.” J. Comput. Civ. Eng. 34 (5). https://doi.org/10.1061/(ASCE)CP.1943-5487.0000918.
Hu, Y., C. Zhao, and H. Wang. 2010. “Automatic pavement crack detection using texture and shape descriptors.” IETE Tech. Rev. 27 (5): 398–405. https://doi.org/10.4103/0256-4602.62225.
Huyan, J., W. Li, S. Tighe, Z. Xu, and J. Zhai. 2020. “CrackU-net: A novel deep convolutional neural network for pixelwise pavement crack detection.” Struct. Control Health Monit. 27 (8): e2551. https://doi.org/10.1002/stc.2551.
Jiang, X., J. Gabrielson, B. Huang, Y. Bai, P. Polaczyk, M. Zhang, W. Hu, and R. Xiao. 2022a. “Evaluation of inverted pavement by structural condition indicators from falling weight deflectometer.” Constr. Build. Mater. 319 (Mar): 125991. https://doi.org/10.1016/j.conbuildmat.2021.125991.
Jiang, X., J. Gabrielson, H. Titi, B. Huang, Y. Bai, P. Polaczyk, W. Hu, M. Zhang, and R. Xiao. 2022b. “Field investigation and numerical analysis of an inverted pavement system in Tennessee, USA.” Transp. Geotech. 35 (Jul): 100759. https://doi.org/10.1016/j.trgeo.2022.100759.
Jiang, X., H. Titi, Y. Ma, P. Polaczyk, M. Zhang, J. Gabrielson, Y. Bai, and B. Huang. 2022c. “Evaluating the performance of inverted pavement structure using the accelerated pavement test (APT).” Constr. Build. Mater. 346 (Mar): 128489. https://doi.org/10.1016/j.conbuildmat.2022.128489.
Kheradmandi, N., and V. Mehranfar. 2022. “A critical review and comparative study on image segmentation-based techniques for pavement crack detection.” Constr. Build. Mater. 321 (Feb): 126162. https://doi.org/10.1016/j.conbuildmat.2021.126162.
Kim, I., H. Jeon, S. Baek, W. Hong, and H. Jung. 2018. “Application of crack identification techniques for an aging concrete bridge inspection using an unmanned aerial vehicle.” Sensors 18 (6): 1881. https://doi.org/10.3390/s18061881.
Kingma, P., and J. Ba. 2015. “Adam: A method for stochastic optimization.” Preprint, submitted December 22, 2014. https://arxiv.org/abs/1412.6980.
Li, X., X. Sun, Y. Meng, J. Liang, F. Wu, and J. Li. 2020. “Dice loss for data-imbalanced NLP tasks.” In Proc., 58th Annual Meeting of the Association for Computational Linguistics, 2020. Stroudsburg, PA: Association for Computational Linguistics.
Liu, F., J. Liu, and L. Wang. 2022a. “Asphalt pavement fatigue crack severity classification by infrared thermography and deep learning.” Autom. Constr. 143 (May): 104575. https://doi.org/10.1016/j.autcon.2022.104575.
Liu, F., J. Liu, and L. Wang. 2022b. “Deep learning and infrared thermography for asphalt pavement crack severity classification.” Autom. Constr. 140 (May): 104383. https://doi.org/10.1016/j.autcon.2022.104383.
Ma, D., H. Fang, N. Wang, C. Zhang, J. Dong, and H. Hu. 2022. “Automatic detection and counting system for pavement cracks based on PCGAN and YOLO-MF.” IEEE Trans. Intell. Transp. Syst. 23 (11): 22166–22178. https://doi.org/10.1109/TITS.2022.3161960.
Majidifard, H., P. Jin, Y. Adu-Gyamfi, and W. Buttlar. 2020. “Pavement image datasets: A new benchmark dataset to classify and densify pavement distresses.” Transp. Res. Rec. 2674 (2): 328–339. https://doi.org/10.1177/0361198120907283.
Mei, Q., and M. Gul. 2020. “A cost effective solution for pavement crack inspection using cameras and deep neural networks.” Constr. Build. Mater. 256 (Feb): 119397. https://doi.org/10.1016/j.conbuildmat.2020.119397.
Oliveira, H., and P. Correia. 2013. “Automatic road crack detection and characterization.” IEEE Trans. Intell. Transp. Syst. 14 (1): 155–168. https://doi.org/10.1109/TITS.2012.2208630.
Pan, Y., X. Zhang, G. Cervone, and L. Yang. 2018. “Detection of asphalt pavement potholes and cracks based on the unmanned aerial vehicle multispectral imagery.” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 11 (10): 3701–3712. https://doi.org/10.1109/JSTARS.2018.2865528.
Shi, Y., L. Cui, Z. Qi, F. Meng, and Z. Chen. 2016. “Automatic road crack detection using random structured forests.” IEEE Trans. Intell. Transp. Syst. 17 (12): 3434–3445. https://doi.org/10.1109/TITS.2016.2552248.
Song, L., and X. Wang. 2021. “Faster region convolutional neural network for automated pavement distress detection.” Road Mater. Pavement Des. 22 (1): 23–41. https://doi.org/10.1080/14680629.2019.1614969.
Song, W., H. Gong, S. Zeng, L. Cong, Y. Sun, H. Wu, and B. Huang. 2021. “Field performance evaluation of open-graded asphalt friction courses: A survival data analysis.” Constr. Build. Mater. 306 (55): 124745. https://doi.org/10.1016/j.conbuildmat.2021.124745.
Song, W., Z. Xu, H. Wu, and Y. Zhan. 2022. “A novel approach to determine mode II fracture toughness of hot mix asphalt.” Theor. Appl. Fract. Mech. 122 (Feb): 103600. https://doi.org/10.1016/j.tafmec.2022.103600.
Spencer, B., V. Hoskere, and Y. Narazaki. 2019. “Advances in computer vision-based civil infrastructure inspection and monitoring.” Engineering 5 (2): 199–222. https://doi.org/10.1016/j.eng.2018.11.030.
Tsai, Y.-C., V. Kaul, and R. M. Mersereau. 2010. “Critical assessment of pavement distress segmentation methods.” J. Transp. Eng. 136 (1): 11–19. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000051.
Xiao, L., W. Li, N. Deng, B. Yuan, Y. Bi, Y. Cui, and X. Cui. 2022. “Automatic pavement crack identification based on an improved C-mask region-based convolutional neural network model.” Transp. Res. Rec. 2677 (3): 1194–1216. https://doi.org/10.1177/03611981221122778.
Yang, F., L. Zhang, S. Yu, D. Prokhorov, X. Mei, and H. Ling. 2020. “Feature pyramid and hierarchical boosting network for pavement crack detection.” IEEE Trans. Intell. Transp. Syst. 21 (4): 1525–1535. https://doi.org/10.1109/TITS.2019.2910595.
Yang, X., H. Li, Y. Yu, X. Luo, T. Huang, and X. Yang. 2018. “Automatic pixel-level crack detection and measurement using fully convolutional network.” Comput.-Aided Civ. Infrastruct. Eng. 33 (12): 1090–1109. https://doi.org/10.1111/mice.12412.
Yao, H., Y. Liu, X. Li, Z. You, Y. Feng, and W. Lu. 2022. “A detection method for pavement cracks combining object detection and attention mechanism.” IEEE Trans. Intell. Transp. Syst. 23 (11): 22179–22189. https://doi.org/10.1109/TITS.2022.3177210.
Zhai, J., Z. Sun, J. Huyan, H. Yang, and W. Li. 2022. “Automatic pavement crack detection using multimodal features fusion deep neural network.” Int. J. Pavement Eng. 1–14. https://doi.org/10.1080/10298436.2022.2086692.
Zhang, A., K. Wang, B. Li, E. Yang, X. Dai, Y. Peng, Y. Fei, Y. Liu, J. Li, and C. Chen. 2017. “Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network.” Comput.-Aided Civ. Infrastruct. Eng. 32 (10): 805–819. https://doi.org/10.1111/mice.12297.
Zhang, A., K. C. P. Wang, Y. Fei, Y. Liu, S. Tao, C. Chen, J. Li, and B. Li. 2018a. “Deep learning—Based fully automated pavement crack detection on 3D asphalt surfaces with an improved CrackNet.” J. Comput. Civ. Eng. 32 (5): 04018041. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000775.
Zhang, Y., B. Chen, J. Wang, J. Li, and X. Sun. 2020. “APLCNet: Automatic pixel-level crack detection network based on instance segmentation.” IEEE Access 8 (Oct): 199159–199170. https://doi.org/10.1109/ACCESS.2020.3033661.
Zhang, Y., C. Chen, Q. Wu, Q. Lu, S. Zhang, G. Zhang, and Y. Yang. 2018b. “A kinect-based approach for 3D pavement surface reconstruction and cracking recognition.” IEEE Trans. Intell. Transp. Syst. 19 (12): 3935–3946. https://doi.org/10.1109/TITS.2018.2791476.
Zhong, J., J. Huyan, W. Zhang, H. Cheng, J. Zhang, Z. Tong, X. Jiang, and B. Huang. 2023. “A deeper generative adversarial network for grooved cement concrete pavement crack detection.” Eng. Appl. Artif. Intell. 119 (Mar): 105808. https://doi.org/10.1016/j.engappai.2022.105808.
Zhong, J., J. Zhu, J. Huyan, T. Ma, and W. Zhang. 2022. “Multi-scale feature fusion network for pixel-level pavement distress detection.” Autom. Constr. 141 (Sep): 104436. https://doi.org/10.1016/j.autcon.2022.104436.
Zhou, Y., F. Wang, N. Meghanathan, and Y. Huang. 2016. “Seed-based approach for automated crack detection from pavement images.” Transp. Res. Rec. 2589 (1): 162–171. https://doi.org/10.3141/2589-18.
Zhu, J., J. Zhong, T. Ma, X. Huang, W. Zhang, and Y. Zhou. 2022. “Pavement distress detection using convolutional neural networks with images captured via UAV.” Autom. Constr. 133 (Jan): 103991. https://doi.org/10.1016/j.autcon.2021.103991.
Information & Authors
Information
Published In
Journal of Transportation Engineering, Part B: Pavements
Volume 150 • Issue 1 • March 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Apr 28, 2023
Accepted: Sep 26, 2023
Published online: Jan 5, 2024
Published in print: Mar 1, 2024
Discussion open until: Jun 5, 2024
ASCE Technical Topics:
- Algorithms
- Artificial intelligence and machine learning
- Automation and robotics
- Computer programming
- Computing in civil engineering
- Engineering fundamentals
- Equipment and machinery
- Gravels
- Infrastructure
- Mathematics
- Neural networks
- Pavement condition
- Pavements
- Systems engineering
- Transportation engineering
- Unmanned vehicles
- Vehicle-pavement interaction
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.