Automatic Pixel-Level Segmentation of Multiple Pavement Distresses and Surface Design Features with PDSNet II
Publication: Journal of Computing in Civil Engineering
Volume 38, Issue 6
Abstract
Effective distress detection and quantitative analysis play a crucial role in road maintenance and driving safety. The Pavement distress segmentation network (PDSNet) is designed to combine the pyramid scene parsing network (PSPNet) and U-Net, providing both prior global information and local features that can overcome the common detection issues on the pavement data set faced by a single network. This paper proposes an efficient and improved architecture of PDSNet called PDSNet II for enhanced global modeling and retrieving fine details capacities. The proposed PDSNet II represents two major modifications on the original PDSNet. Firstly, a shifted window based on fully connected conditional random fields (FC-CRFs) layer is purposefully introduced to provide connections among consecutive self-attention layers that significantly enhance modeling power. Secondly, PDSNet II adopts multiple-head attention mechanisms to capture diverse interaction information across multiple projection spaces. Consequently, the output maps from the pyramid pooling module (PPM) head and the U-Net tail are fed into a neural window FC-CRFs layer. PDSNet II was trained using a data set consisting of 12,648 two-dimensional (2D) intensity and three-dimensional (3D) range images depicting various pavement conditions. The experimental results demonstrate that PDSNet II outperforms the original PDSNet in terms of F1-score and intersection over union (IoU). Compared with state-of-the-art networks, PDSNet II exhibits superior performance in detecting complex distress patterns, while effectively reducing noise and maintaining robustness. Overall, the proposed PDSNet II framework shows promising results in pavement distress segmentation, highlighting its potential for practical applications.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code used during the study were provided by a third party. Direct requests for these materials may be made to the provider as indicated in the Acknowledgements. The testing data set is firstly publicly available at https://github.com/IP2RG/PDSNet-II-TEST.
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant Nos. 52372334 and 62206201, the Key Research and Development Program of Yunnan Province under Grant No. 202303AA080016, and the China Postdoctoral Science Foundation under Grant No. 2023M732644.
References
Canchila, C., S. Zhou, and W. Song. 2024. “Hyperparameter optimization and importance ranking in deep learning–based crack segmentation.” J. Comput. Civ. Eng. 38 (2): 04023042. https://doi.org/10.1061/JCCEE5.CPENG-5512.
Chen, F.-C., M. R. Jahanshahi, R.-T. Wu, and C. Joffe. 2017. “A texture-based video processing methodology using Bayesian data fusion for autonomous crack detection on metallic surfaces.” Comput.-Aided Civ. Infrastruct. Eng. 32 (4): 271–287. https://doi.org/10.1111/mice.12256.
Chen, T., Z. Cai, X. Zhao, C. Chen, X. Liang, T. Zou, and P. Wang. 2020. “Pavement crack detection and recognition using the architecture of segNet.” J. Ind. Inf. Integr. 18 (Jun): 100144. https://doi.org/10.1016/j.jii.2020.100144.
Chou, J., W. A. O’Neill, and H. Cheng. 1994. “Pavement distress classification using neural networks.” In Proc., IEEE Int. Conf. on Systems, Man and Cybernetics, 397–401. New York: IEEE.
Cubero-Fernandez, A., F. Rodriguez-Lozano, R. Villatoro, J. Olivares, and J. M. Palomares. 2017. “Efficient pavement crack detection and classification.” EURASIP J. Image Video Process. 2017 (Dec): 1–11. https://doi.org/10.1186/s13640-017-0187-0.
Defang, W., and W. N. Zeng Weiming. 2015. “Road crack detection under uneven illuminationîĂČ using improved k-means algorithm.” Comput. Appl. Software 32 (7): 244–247. https://doi.org/10.1080/10298436.2023.2246097.
Ding, S., H. Lang, J. Chen, Y. Yuan, and J. Lu. 2023. “Automated crack segmentation on 3D asphalt surfaces with richer attention and hybrid pyramid structures.” Int. J. Pavement Eng. 24 (1): 2246097. https://doi.org/10.1080/10298436.2023.2246097.
Ding, S., Y. Xing, H. Lang, T. Wen, and J. J. Lu. 2022. “Line-structured light rut detection of asphalt pavement with pavement markings interference under strong light.” J. Transp. Eng. Part B: Pavements 148 (2): 04022007. https://doi.org/10.1061/JPEODX.0000341.
Dosovitskiy, A., et al. 2020. “An image is worth 16x16 words: Transformers for image recognition at scale.” Preprint, submitted October 22, 2020. https://arxiv.org/abs/2010.11929.
Fan, R., M. J. Bocus, Y. Zhu, J. Jiao, L. Wang, F. Ma, S. Cheng, and M. Liu. 2019. “Road crack detection using deep convolutional neural network and adaptive thresholding.” In Proc., 2019 IEEE Intelligent Vehicle Symp., 474–479. New York: IEEE.
Fei, Y., K. C. Wang, A. Zhang, C. Chen, J. Q. Li, Y. Liu, G. Yang, and B. Li. 2019. “Pixel-level cracking detection on 3D asphalt pavement images through deep-learning-based CrackNet-V.” IEEE Intell. Transp. Syst. 21 (1): 273–284. https://doi.org/10.1109/TITS.2019.2891167.
Fu, J., J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu. 2019. “Dual attention network for scene segmentation.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 3146–3154. New York: IEEE.
Guo, F., Y. Qian, J. Liu, and H. Yu. 2023. “Pavement crack detection based on transformer network.” Autom. Constr. 145 (Jan): 104646. https://doi.org/10.1016/j.autcon.2022.104646.
Hsieh, Y.-A., and Y. J. Tsai. 2020. “Machine learning for crack detection: Review and model performance comparison.” J. Comput. Civ. Eng. 34 (5): 04020038. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000918.
Hua, Y., and H. Tian. 2016. “Depth estimation with convolutional conditional random field network.” Neurocomputing 214 (Nov): 546–554. https://doi.org/10.1016/j.neucom.2016.06.029.
Huang, J., W. Liu, and X. Sun. 2014. “A pavement crack detection method combining 2D with 3D information based on Dempster-Shafer theory.” Comput.-Aided Civ. Infrastruct. Eng. 29 (4): 299–313. https://doi.org/10.1111/mice.12041.
IP2RG. 2024. “PDSNet-II-TEST.” Accessed December 18, 2023. https://github.com/IP2RG/PDSNet-II-TEST.
Islam, M. M., and J.-M. Kim. 2019. “Vision-based autonomous crack detection of concrete structures using a fully convolutional encoder–decoder network.” Sensors 19 (19): 4251. https://doi.org/10.3390/s19194251.
Ji, A., X. Xue, Y. Wang, X. Luo, and W. Xue. 2020. “An integrated approach to automatic pixel-level crack detection and quantification of asphalt pavement.” Autom. Constr. 114 (Jun): 103176. https://doi.org/10.1016/j.autcon.2020.103176.
Kalfarisi, R., Z. Y. Wu, and K. Soh. 2020. “Crack detection and segmentation using deep learning with 3D reality mesh model for quantitative assessment and integrated visualization.” J. Comput. Civ. Eng. 34 (3): 04020010. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000890.
Kapur, J. N., P. K. Sahoo, and A. K. Wong. 1985. “A new method for gray-level picture thresholding using the entropy of the histogram.” Comput. Vision Graphics Image Process. 29 (3): 273–285. https://doi.org/10.1016/0734-189X(85)90125-2.
Kaseko, M. S., Z.-P. Lo, and S. G. Ritchie. 1994. “Comparison of traditional and neural classifiers for pavement-crack detection.” J. Transp. Eng. 120 (4): 552–569. https://doi.org/10.1061/(ASCE)0733-947X(1994)120:4(552).
Kirschke, K., and S. Velinsky. 1992. “Histogram-based approach for automated pavement-crack sensing.” J. Transp. Eng. 118 (5): 700–710. https://doi.org/10.1061/(ASCE)0733-947X(1992)118:5(700).
Lang, H., J. J. Lu, Y. Lou, and S. Chen. 2020. “Pavement cracking detection and classification based on 3D image using multiscale clustering model.” J. Comput. Civ. Eng. 34 (5): 04020034. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000910.
Lee, B. J., and H. Â. Lee. 2004. “Position-invariant neural network for digital pavement crack analysis.” Comput.-Aided Civ. Infrastruct. Eng. 19 (2): 105–118. https://doi.org/10.1111/j.1467-8667.2004.00341.x.
Li, P., B. Zhou, C. Wang, G. Hu, Y. Yan, R. Guo, and H. Xia. 2024. “CNN-based pavement defects detection using grey and depth images.” Autom. Constr. 158 (Feb): 105192. https://doi.org/10.1016/j.autcon.2023.105192.
Li, S., Y. Cao, and H. Cai. 2017. “Automatic pavement-crack detection and segmentation based on steerable matched filtering and an active contour model.” J. Comput. Civ. Eng. 31 (5): 04017045. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000695.
Lin, T.-Y., P. Goyal, R. Girshick, K. He, and P. Dollár. 2017. “Focal loss for dense object detection.” In Proc., IEEE Int. Conf. on Computation Vision, 2980–2988. New York: IEEE.
Liu, Z., Y. Cao, Y. Wang, and W. Wang. 2019. “Computer vision-based concrete crack detection using U-Net fully convolutional networks.” Autom. Constr. 104 (Aug): 129–139. https://doi.org/10.1016/j.autcon.2019.04.005.
Mathavan, S., M. Rahman, and K. Kamal. 2015. “Use of a self-organizing map for crack detection in highly textured pavement images.” J. Infrastruct. Syst. 21 (3): 04014052. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000237.
MOT (Ministry of Transport of the People’s Republic of China). 2019. Highway performance assessment standards. Beijing: MOT.
Peng, Z., W. Huang, S. Gu, L. Xie, Y. Wang, J. Jiao, and Q. Ye. 2021. “Conformer: Local features coupling global representations for visual recognition.” In Proc., IEEE/CVF Int. Conf. on Computation Vision, 367–376. New York: IEEE.
Pham, M.-V., Y.-S. Ha, and Y.-T. Kim. 2023. “Automatic detection and measurement of ground crack propagation using deep learning networks and an image processing technique.” Measurement 215 (Jun): 112832. https://doi.org/10.1016/j.measurement.2023.112832.
Qu, Z., L. Bai, S.-Q. An, F.-R. Ju, and L. Liu. 2016. “Lining seam elimination algorithm and surface crack detection in concrete tunnel lining.” J. Electron. Imaging 25 (6): 063004. https://doi.org/10.1117/1.JEI.25.6.063004.
Ricci, E., W. Ouyang, X. Wang, and N. Sebe. 2018. “Monocular depth estimation using multi-scale continuous CRFS as sequential deep networks.” IEEE Trans. Pattern Anal. Mach. Intell. 41 (6): 1426–1440. https://doi.org/10.1109/TPAMI.2018.2839602.
Ronneberger, O., P. Fischer, and T. Brox. 2015. “U-net: Convolutional networks for biomedical image segmentation.” In Proc., 18th Int. Conf. on Medical Image Computing Computer-Assisted Intervention, 234–241. Berlin: Springer.
Shang, J., J. Xu, A. A. Zhang, Y. Liu, K. C. Wang, D. Ren, H. Zhang, Z. Dong, and A. He. 2023. “Automatic pixel-level pavement sealed crack detection using multi-fusion U-Net network.” Measurement 208 (Feb): 112475. https://doi.org/10.1016/j.measurement.2023.112475.
Shi, Y., L. Cui, Z. Qi, F. Meng, and Z. Chen. 2016. “Automatic road crack detection using random structured forests.” IEEE Intell. Transp. Syst. 17 (12): 3434–3445. https://doi.org/10.1109/TITS.2016.2552248.
Sollazzo, G., K. C. Wang, G. Bosurgi, and J. Q. Li. 2016. “Hybrid procedure for automated detection of cracking with 3D pavement data.” J. Comput. Civ. Eng. 30 (6): 04016032. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000597.
Tsai, Y.-C., and A. Chatterjee. 2018. “Pothole detection and classification using 3D technology and watershed method.” J. Comput. Civ. Eng. 32 (2): 04017078. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000726.
Tsai, Y.-C., V. Kaul, and R. M. Mersereau. 2010. “Critical assessment of pavement distress segmentation methods.” J. Transp. Eng. 136 (1): 11–19. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000051.
Wang, A., H. Lang, Z. Chen, Y. Peng, S. Ding, and J. J. Lu. 2024. “The two-step method of pavement pothole and raveling detection and segmentation based on deep learning.” In Proc., IEEE Transactions on Intelligent Transportation Systems. New York: IEEE.
Wang, L., L. Zhuang, and Z. Zhang. 2019. “Automatic detection of rail surface cracks with a superpixel-based data-driven framework.” J. Comput. Civ. Eng. 33 (1): 04018053. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000799.
Wang, S., S. Qiu, W. Wang, D. Xiao, and K. C. Wang. 2017. “Cracking classification using minimum rectangular cover–based support vector machine.” J. Comput. Civ. Eng. 31 (5): 04017027. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000672.
Wen, T., S. Ding, H. Lang, J. J. Lu, Y. Yuan, Y. Peng, J. Chen, and A. Wang. 2022a. “Automated pavement distress segmentation on asphalt surfaces using a deep learning network.” Int. J. Pavement. Eng. 24 (2): 1–14. https://doi.org/10.1080/10298436.2022.2027414.
Wen, T., H. Lang, S. Ding, J. J. Lu, and Y. Xing. 2022b. “PCDNet: Seed operation–based deep learning model for pavement crack detection on 3D asphalt surface.” J. Transp. Eng. Part B: Pavements 148 (2): 04022023. https://doi.org/10.1061/JPEODX.0000367.
Xie, E., W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. 2021. “Segformer: Simple and efficient design for semantic segmentation with transformers.” Adv. Neural Inf. Process. Syst. 34 (Dec): 12077–12090.
Xu, W., Z. Tang, J. Zhou, and J. Ding. 2013. “Pavement crack detection based on saliency and statistical features.” In Proc., 2013 IEEE Int. Conf. on Image Processing, 4093–4097. New York: IEEE.
Yuan, W., X. Gu, Z. Dai, S. Zhu, and P. Tan. 2022. “New CRFS: Neural window fully-connected CRFS for monocular depth estimation.” Preprint, submitted March 3, 2022. https://arxiv.org/abs/2203.01502.
Zhang, A., K. C. Wang, Y. Fei, Y. Liu, C. Chen, G. Yang, J. Q. Li, E. Yang, and S. Qiu. 2019. “Automated pixel-level pavement crack detection on 3d asphalt surfaces with a recurrent neural network.” Comput.-Aided Civ. Infrastruct. Eng 34 (3): 213–229. https://doi.org/10.1111/mice.12409.
Zhang, A., K. C. Wang, Y. Fei, Y. Liu, S. Tao, C. Chen, J. Q. Li, and B. Li. 2018a. “Deep learning–based fully automated pavement crack detection on 3D asphalt surfaces with an improved cracknet.” J. Comput. Civ. Eng., 32 (5): 04018041. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000775.
Zhang, A. A., et al. 2022. “Intelligent pixel-level detection of multiple distresses and surface design features on asphalt pavements.” Comput.-Aided Civ. Infrastruct. Eng. 37 (13): 1654–1673. https://doi.org/10.1111/mice.12909.
Zhang, H., A. A. Zhang, Z. Dong, A. He, Y. Liu, Y. Zhan, and K. C. Wang. 2024. “Robust semantic segmentation for automatic crack detection within pavement images using multi-mixing of global context and local image features.” In Proc., IEEE Transactions on Intelligent Transportation Systems. New York: IEEE.
Zhang, Y., C. Chen, Q. Wu, Q. Lu, S. Zhang, G. Zhang, and Y. Yang. 2018b. “A kinect-based approach for 3D pavement surface reconstruction and cracking recognition.” IEEE Intell. Transp. Syst. 19 (12): 3935–3946. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000597.
Zhao, H., J. Shi, X. Qi, X. Wang, and J. Jia. 2017. “Pyramid scene parsing network.” In Proc., IEEE Conf. on Computing Vision Pattern Recognition, 2881–2890. New York: IEEE.
Zhou, S., and W. Song. 2021. “Crack segmentation through deep convolutional neural networks and heterogeneous image fusion.” Autom. Constr. 125 (May): 103605. https://doi.org/10.1016/j.autcon.2021.103605.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 38 • Issue 6 • November 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Dec 19, 2023
Accepted: Apr 1, 2024
Published online: Jul 26, 2024
Published in print: Nov 1, 2024
Discussion open until: Dec 26, 2024
ASCE Technical Topics:
- Automation and robotics
- Business management
- Design (by type)
- Engineering fundamentals
- Gravels
- Highway and road design
- Highway engineering
- Highway transportation
- Infrastructure
- Intelligent transportation systems
- Pavement condition
- Pavement design
- Pavements
- Practice and Profession
- Public administration
- Public health and safety
- Safety
- Sight distances
- Structural engineering
- Structural systems
- Systems engineering
- Traffic engineering
- Traffic management
- Traffic safety
- Transportation engineering
- Transportation management
- Windows
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.